Nov 01 20:30:03.496201 ip-10-0-133-242 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:30:03.921120 ip-10-0-133-242 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:03.921120 ip-10-0-133-242 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:30:03.921120 ip-10-0-133-242 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:03.921120 ip-10-0-133-242 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:03.927036 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.926953 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:03.927036 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927031 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:03.927036 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927036 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:03.927036 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927039 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:03.927036 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927043 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927047 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927049 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927052 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927054 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927057 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927060 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927062 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927064 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927067 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927070 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927072 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927075 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927077 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927079 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927082 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927084 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927087 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927089 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927091 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:03.927225 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927094 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927100 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927104 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927110 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927113 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927116 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927119 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927122 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927125 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927127 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927130 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927133 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927135 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927139 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927141 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927144 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927147 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927151 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927155 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:03.927718 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927158 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927160 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927163 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927165 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927168 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927170 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927173 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927176 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927179 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927181 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927184 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927186 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927188 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927191 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927193 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927196 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927198 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927201 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927203 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927205 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:03.928176 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927208 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927211 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927214 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927216 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927218 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927221 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927224 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927227 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927230 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927232 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927235 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927238 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927241 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927243 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927246 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927248 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927250 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927253 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927255 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927258 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:03.928660 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927261 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927263 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927266 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927268 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927270 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927273 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927275 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927278 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927280 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927283 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927285 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927287 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927290 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927292 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927296 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927301 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927303 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:03.929141 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927309 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:03.929595 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927312 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:03.929595 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927314 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:03.929595 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.927317 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:03.930476 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930462 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:03.930476 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930476 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930480 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930483 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930486 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930489 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930491 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930494 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930497 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930500 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930503 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930505 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930508 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930511 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930513 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930516 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930518 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930520 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930523 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930525 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930528 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:03.930529 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930530 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930534 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930538 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930542 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930545 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930547 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930550 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930553 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930555 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930558 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930560 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930563 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930565 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930568 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930570 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930573 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930575 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930577 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:03.931179 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930580 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930583 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930586 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930588 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930591 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930594 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930596 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930598 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930601 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930603 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930606 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930608 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930610 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930613 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930615 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930618 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930621 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930623 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930626 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930629 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:03.931619 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930632 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930636 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930639 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930642 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930644 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930647 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930649 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930652 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930654 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930657 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930659 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930662 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930664 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930666 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930669 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930671 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930674 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930678 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930680 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930683 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:03.932139 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930685 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930688 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930690 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930693 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930695 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930697 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930700 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930703 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930706 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930709 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930713 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930715 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930718 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930720 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930723 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930726 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930728 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930730 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930733 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930735 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:03.932612 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930738 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930740 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930743 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930745 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930748 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930751 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.930753 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931302 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931309 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931313 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931316 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931319 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931321 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931324 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931327 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931330 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931332 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931335 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931337 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931339 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:03.933107 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931343 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931346 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931348 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931351 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931353 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931356 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931358 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931361 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931363 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931366 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931368 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931371 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931373 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931375 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931378 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931382 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931386 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931389 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931391 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:03.933585 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931394 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931396 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931399 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931402 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931404 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931407 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931409 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931411 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931414 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931416 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931419 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931421 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931423 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931427 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931429 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931432 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931434 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931437 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931439 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931441 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:03.934050 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931446 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931449 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931452 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931455 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931457 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931459 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931462 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931464 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931467 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931470 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931472 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931475 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931477 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931480 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931483 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931485 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931488 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931491 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931493 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931496 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:03.934525 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931498 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931501 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931503 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931505 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931508 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931510 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931513 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931515 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931518 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931521 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931523 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931526 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931528 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931531 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931533 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931535 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931538 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931540 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931543 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:03.935003 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931545 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931547 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931551 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931553 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931555 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931558 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931561 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931563 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931566 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931568 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931571 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931573 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931576 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931578 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.931582 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932532 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932549 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932555 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932560 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932564 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:30:03.935454 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932567 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932572 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932576 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932579 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932582 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932585 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932589 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932592 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932594 2578 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932597 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932600 2578 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932603 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932606 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932611 2578 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932615 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932618 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932621 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932625 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932629 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932631 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932634 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932637 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932640 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932644 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:30:03.935934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932647 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932650 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932653 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932657 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932660 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932663 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932666 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932669 2578 flags.go:64] FLAG: --enable-server="true" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932672 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932676 2578 flags.go:64] FLAG: --event-burst="100" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932679 2578 flags.go:64] FLAG: --event-qps="50" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932682 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932685 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932687 2578 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932692 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932695 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932698 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932701 2578 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932704 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932707 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932710 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932713 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932716 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932718 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932721 2578 flags.go:64] FLAG: --feature-gates="" Nov 01 20:30:03.937176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932725 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932728 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932731 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932734 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932737 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932740 2578 flags.go:64] FLAG: --help="false" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932743 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932747 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932750 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932753 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932756 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932760 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932765 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932768 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932770 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932773 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932776 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932779 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932782 2578 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932785 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932788 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932791 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932793 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:30:03.937788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932796 2578 flags.go:64] FLAG: --lock-file="" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932799 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932802 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932805 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932810 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932813 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932816 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932819 2578 flags.go:64] FLAG: --logging-format="text" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932822 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932825 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932828 2578 flags.go:64] FLAG: --manifest-url="" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932830 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932835 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932850 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932855 2578 flags.go:64] FLAG: --max-pods="110" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932858 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932861 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932864 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932867 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932870 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932873 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932877 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932884 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932887 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932890 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:30:03.938369 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932893 2578 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932896 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932900 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932902 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932905 2578 flags.go:64] FLAG: --port="10250" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932908 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932911 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-017a2b02feea7ba52" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932914 2578 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932917 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932920 2578 flags.go:64] FLAG: --register-node="true" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932923 2578 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932927 2578 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932931 2578 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932934 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932937 2578 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932941 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932944 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932947 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932950 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932952 2578 flags.go:64] FLAG: --runonce="false" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932955 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932959 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932961 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932964 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932967 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932970 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:30:03.938987 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932973 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932976 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932980 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932983 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932986 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932989 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932992 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932994 2578 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.932997 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933002 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933005 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933008 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933013 2578 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933016 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933018 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933021 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933024 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933027 2578 flags.go:64] FLAG: --v="2" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933033 2578 flags.go:64] FLAG: --version="false" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933037 2578 flags.go:64] FLAG: --vmodule="" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933041 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933044 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933190 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933195 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:03.939620 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933198 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933200 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933203 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933206 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933209 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933211 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933214 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933216 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933225 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933228 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933232 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933234 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933237 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933240 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933243 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933245 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933248 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933250 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933253 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933256 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:03.940209 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933258 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933261 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933263 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933266 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933268 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933271 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933275 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933277 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933280 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933282 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933285 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933288 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933290 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933295 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933298 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933300 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933303 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933305 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:03.940721 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933308 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933310 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933314 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933318 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933320 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933323 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933325 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933328 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933331 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933333 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933336 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933339 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933342 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933344 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933347 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933349 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933352 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933354 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933357 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:03.941199 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933361 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933363 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933366 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933368 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933371 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933373 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933376 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933378 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933381 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933383 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933386 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933388 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933391 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933393 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933396 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933398 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933403 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933406 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933408 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933411 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:03.941655 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933414 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933416 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933419 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933421 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933424 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933426 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933429 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933431 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933434 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933436 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933439 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933442 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933446 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933450 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933453 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933455 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933459 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933463 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933466 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:03.942134 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933469 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933471 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933474 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933476 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933479 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933481 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933484 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.933509 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933549 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933554 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933559 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933562 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933565 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933568 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933571 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933573 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:03.942581 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933576 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933579 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933581 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933584 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933586 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933589 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933592 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933594 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933597 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933601 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933604 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933606 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933609 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933611 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933614 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933616 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933619 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933622 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933624 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:03.942974 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933627 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933630 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933632 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933635 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933637 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933640 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933642 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933646 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933654 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933657 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933660 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933662 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933665 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933667 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933670 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933672 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933674 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933677 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933679 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:03.943423 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933682 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933684 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933687 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933690 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933693 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933695 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933698 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933700 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933702 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933705 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933708 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933710 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933712 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933715 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933717 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933720 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933722 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933725 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933727 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933730 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:03.943881 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933733 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933736 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933741 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933745 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933747 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933751 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933754 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933757 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933761 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933764 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933767 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933770 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933772 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933775 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933777 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933781 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933783 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933786 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933788 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:03.944360 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933791 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933793 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933796 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933798 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933801 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933803 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933805 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933808 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933810 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933812 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933815 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933817 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933820 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933824 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933827 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933829 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933835 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933851 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933854 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933856 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:03.944800 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.933859 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.941181 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.941196 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.941212 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.941217 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941278 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941282 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941285 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941288 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941290 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941293 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941296 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941299 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941302 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941304 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941307 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941309 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941312 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941314 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941317 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:03.945298 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941319 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941322 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941324 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941327 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941329 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941332 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941334 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941337 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941340 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941342 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941344 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941347 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941349 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941352 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941356 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941358 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941363 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941367 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941370 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:03.945772 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941373 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941376 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941378 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941381 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941384 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941386 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941390 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941394 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941397 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941399 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941402 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941404 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941407 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941409 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941412 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941415 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941417 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941420 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941422 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:03.946259 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941425 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941428 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941430 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941432 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941435 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941437 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941440 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941442 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941446 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941449 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941451 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941454 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941456 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941459 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941461 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941464 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941466 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941469 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941471 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941474 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:03.946706 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941476 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941478 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941481 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941483 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941485 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941488 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941491 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941493 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941496 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941498 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941500 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941503 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941505 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941508 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941510 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941513 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941515 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941517 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941520 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941522 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:03.947258 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941525 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941528 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941531 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941533 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941536 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941538 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941541 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941543 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941546 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941548 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941551 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941553 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941555 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.941582 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941773 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941778 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:03.947738 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941781 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941784 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941787 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941790 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941793 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941795 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941798 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941800 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941804 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941807 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941809 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941814 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941818 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941821 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941824 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941827 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941829 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941832 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:03.948121 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941835 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941837 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941855 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941858 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941861 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941864 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941867 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941869 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941872 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941874 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941877 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941879 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941882 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941884 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941886 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941889 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941891 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941894 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941897 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941899 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:03.948785 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941902 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941904 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941907 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941909 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941911 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941914 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941917 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941919 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941922 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941925 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941927 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941930 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941932 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941935 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941938 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941941 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941943 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941946 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941948 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941951 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:03.949599 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941953 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941955 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941959 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941963 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941965 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941968 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941970 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941973 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941975 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941977 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941980 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941982 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941984 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941987 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941990 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941992 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941995 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.941997 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942000 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942002 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:03.950214 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942005 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942007 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942010 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942012 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942015 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942017 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942020 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942022 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942025 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942027 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942029 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942032 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942034 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942037 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942039 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942041 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942044 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942046 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942049 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:03.950955 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942051 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942053 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942056 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942058 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942061 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942063 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.942086 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942131 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942136 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942139 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942142 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942145 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942147 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942150 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942152 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942154 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:03.951503 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942157 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942159 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942162 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942164 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942174 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942177 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942179 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942182 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942184 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942186 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942189 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942191 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942193 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942196 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942198 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942202 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942206 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942209 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942211 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:03.952150 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942214 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942216 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942218 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942221 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942223 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942226 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942228 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942231 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942233 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942236 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942240 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942242 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942244 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942247 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942249 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942252 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942254 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942256 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942264 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942267 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:03.952604 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942270 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942272 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942275 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942277 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942279 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942282 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942287 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942289 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942292 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942294 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942297 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942299 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942301 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942304 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942306 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942308 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942311 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942313 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942316 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942318 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:03.953111 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942320 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942323 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942325 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942327 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942331 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942334 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942337 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942339 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942342 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942344 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942346 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942356 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942359 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942362 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942364 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942367 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942369 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942371 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942375 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:03.953580 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942378 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942381 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942383 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942386 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942388 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942390 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942394 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942396 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942400 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942402 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942405 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942407 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942410 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942412 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942415 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942417 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942419 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942422 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:03.942424 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:03.954077 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.942482 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:30:03.954545 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.944677 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 20:30:03.954545 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.945636 2578 server.go:1014] "Starting client certificate rotation" Nov 01 20:30:03.954545 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.945747 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:30:03.954545 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.946341 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:30:03.954545 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.947737 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:30:03.954545 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.952106 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:30:03.964244 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.964225 2578 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:30:03.968357 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.968334 2578 log.go:25] "Validated CRI v1 image API" Nov 01 20:30:03.969256 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.969231 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:30:03.974722 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.974699 2578 fs.go:135] Filesystem UUIDs: map[34fea475-01f9-4aee-ad32-a87247e2cf85:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8451124f-b9d2-4b9f-988f-69cf9d22cf89:/dev/nvme0n1p3] Nov 01 20:30:03.974791 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.974722 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 20:30:03.979088 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.978975 2578 manager.go:217] Machine: {Timestamp:2026-11-01 20:30:03.978466363 +0000 UTC m=+0.358704680 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098520 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24cd7457da3e6b238f00f217bf14a0 SystemUUID:ec24cd74-57da-3e6b-238f-00f217bf14a0 BootID:4dcfa89e-c429-44ac-acc4-42c1bae4a3a9 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:1d:8b:88:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:1d:8b:88:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:b7:47:a9:3e:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:30:03.979827 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.979816 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:30:03.979930 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.979917 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:30:03.984654 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.984625 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:30:03.984796 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.984656 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-242.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:30:03.984862 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.984798 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:30:03.984862 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.984807 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:30:03.984862 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.984836 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:30:03.985618 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.985607 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:30:03.986741 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.986731 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:30:03.990772 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.990760 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:30:03.990915 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.990906 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:30:03.991643 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.991634 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:30:03.991684 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.991648 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:30:03.991684 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.991658 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:30:03.991684 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.991669 2578 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:30:03.991684 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.991677 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:30:03.992188 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.992159 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:30:03.994481 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:03.994461 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:30:04.001562 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.001544 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:30:04.002414 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.002402 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:30:04.002455 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.002432 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:30:04.002610 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.002600 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:30:04.002649 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.002617 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:30:04.002649 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.002623 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:30:04.002649 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.002629 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:30:04.002649 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.002634 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:30:04.002649 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.002640 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:30:04.003267 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.003258 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:30:04.003267 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.003267 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:30:04.003325 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.003275 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:30:04.003325 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.003281 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:30:04.003325 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.003298 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:30:04.003325 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.003307 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:30:04.004236 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.004226 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:30:04.004236 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.004236 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:30:04.008377 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.008354 2578 server.go:1263] "Started kubelet" Nov 01 20:30:04.008484 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.008436 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:30:04.008560 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.008446 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:30:04.008608 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.008589 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:30:04.009122 ip-10-0-133-242 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:30:04.012082 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.012058 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:30:04.015470 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.015456 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:30:04.016952 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.016935 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 20:30:04.020043 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.020028 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:30:04.020554 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.020537 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:30:04.020671 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.020654 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:04.021226 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.021202 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:30:04.021311 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.021236 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:30:04.021311 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.021246 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:30:04.021395 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.021335 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:30:04.021395 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.021346 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:30:04.021810 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.021792 2578 factory.go:55] Registering systemd factory Nov 01 20:30:04.021914 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.021813 2578 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:30:04.022071 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.022061 2578 factory.go:153] Registering CRI-O factory Nov 01 20:30:04.022105 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.022072 2578 factory.go:223] Registration of the crio container factory successfully Nov 01 20:30:04.022169 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.022123 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:30:04.022199 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.022178 2578 factory.go:103] Registering Raw factory Nov 01 20:30:04.022199 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.022195 2578 manager.go:1199] Started watching for new ooms in manager Nov 01 20:30:04.023058 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.023045 2578 manager.go:319] Starting recovery of all containers Nov 01 20:30:04.032913 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.032899 2578 manager.go:324] Recovery completed Nov 01 20:30:04.033976 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.033958 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 01 20:30:04.037107 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.037088 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:04.039811 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.039791 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:04.039896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.039827 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:04.039896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.039862 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:04.040430 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.040415 2578 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:30:04.040430 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.040429 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:30:04.040539 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.040448 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:30:04.040539 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.040472 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:04.041856 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.041831 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:04.041978 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.041969 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:04.042020 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.041981 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:04.042450 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.042439 2578 policy_none.go:50] "Start" Nov 01 20:30:04.042481 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.042454 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:30:04.042481 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.042464 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:30:04.044397 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.044386 2578 policy_none.go:44] "Start" Nov 01 20:30:04.081129 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.081098 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 20:30:04.083373 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.083358 2578 manager.go:341] "Starting Device Plugin manager" Nov 01 20:30:04.083458 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.083393 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 20:30:04.083458 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.083405 2578 server.go:87] "Starting device plugin registration server" Nov 01 20:30:04.083690 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.083677 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 20:30:04.083749 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.083691 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 20:30:04.083792 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.083774 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 20:30:04.083897 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.083883 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 20:30:04.083946 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.083897 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 20:30:04.084596 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.084456 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 20:30:04.084705 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.084620 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:04.147290 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.147261 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-242.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 20:30:04.147290 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.147259 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-242.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 20:30:04.149359 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.147261 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-242.us-west-1.compute.internal.18e405de517a1b93 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-242.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-242.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:04.008135571 +0000 UTC m=+0.388373888,LastTimestamp:2026-11-01 20:30:04.008135571 +0000 UTC m=+0.388373888,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-242.us-west-1.compute.internal,}" Nov 01 20:30:04.154634 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.154598 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:30:04.155989 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.155969 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:30:04.155989 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.155990 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:30:04.156132 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.156009 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:30:04.156132 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.156040 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 20:30:04.184727 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.184640 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:04.185606 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.185586 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:04.185718 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.185618 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:04.185718 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.185629 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:04.185718 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.185654 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.204732 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.204471 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 20:30:04.204732 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.204508 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-242.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 20:30:04.204732 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.204566 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-242.us-west-1.compute.internal.18e405de535d6eee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-242.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-242.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-242.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:04.039810798 +0000 UTC m=+0.420049114,LastTimestamp:2026-11-01 20:30:04.039810798 +0000 UTC m=+0.420049114,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-242.us-west-1.compute.internal,}" Nov 01 20:30:04.221052 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.221024 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 20:30:04.221156 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.221089 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sjs9s" Nov 01 20:30:04.256372 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.256339 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal"] Nov 01 20:30:04.256549 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.256403 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:04.257422 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.257406 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:04.257485 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.257436 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:04.257485 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.257450 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:04.259737 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.259725 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:04.259898 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.259883 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.259981 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.259919 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:04.260478 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.260461 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:04.260555 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.260490 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:04.260555 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.260491 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:04.260555 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.260507 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:04.260555 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.260513 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:04.260555 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.260527 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:04.262728 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.262709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.262828 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.262741 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:04.263502 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.263485 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:04.263571 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.263516 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:04.263571 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.263526 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:04.295751 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.295734 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.298269 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.298249 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.301642 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.301624 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:04.302589 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.302576 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:04.323663 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.323635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.323794 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.323671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.323794 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.323693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb754e3320d8dad235549d1c4b2afdd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"fb754e3320d8dad235549d1c4b2afdd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.351384 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.351357 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.351384 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.351384 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-242.us-west-1.compute.internal\": node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:04.424464 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.424431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.424464 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.424461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.424683 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.424479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb754e3320d8dad235549d1c4b2afdd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"fb754e3320d8dad235549d1c4b2afdd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.424683 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.424528 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb754e3320d8dad235549d1c4b2afdd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"fb754e3320d8dad235549d1c4b2afdd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.424683 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.424543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.424683 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.424560 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.465920 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.465825 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:04.566539 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.566498 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:04.598103 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.598077 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.601162 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.601141 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:04.667202 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.667166 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:04.767570 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.767508 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:04.867976 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.867952 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:04.945302 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.945282 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 20:30:04.945716 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.945392 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 20:30:04.945716 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:04.945451 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 20:30:04.945716 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.945447 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9584bc60fd4645d0a6dd8d98805fda7-6263b06240bad7c4.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-242.us-west-1.compute.internal.18e405de535dcbb2\": read tcp 10.0.133.242:56014->18.224.125.164:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-242.us-west-1.compute.internal.18e405de535dcbb2 default 4995 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-242.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-242.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-242.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:04 +0000 UTC,LastTimestamp:2026-11-01 20:30:04.257441617 +0000 UTC m=+0.637679933,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-242.us-west-1.compute.internal,}" Nov 01 20:30:04.968636 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:04.968608 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:05.069429 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:05.069357 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:05.169731 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:05.169699 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:05.217958 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.217931 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:05.219019 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.219002 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:05.219109 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.219036 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:05.219109 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.219051 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:05.225749 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.225719 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 20:25:05 +0000 UTC" deadline="2028-08-06 00:27:57.965543544 +0000 UTC" Nov 01 20:30:05.225749 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.225746 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15435h57m52.739800276s" Nov 01 20:30:05.270815 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:05.270783 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:05.351627 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.351550 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:05.401477 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.401447 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:05.421517 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.421488 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:05.481276 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.481253 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:05.482204 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.482190 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:05.539065 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:05.539039 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:07.119822 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.119792 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4758d5c314bffde2671732f370e32950.slice/crio-810b2f22f38bcd830eed1afa324fc0e18b6e9f12cc8ee5a95e2d4e64c1215f54 WatchSource:0}: Error finding container 810b2f22f38bcd830eed1afa324fc0e18b6e9f12cc8ee5a95e2d4e64c1215f54: Status 404 returned error can't find the container with id 810b2f22f38bcd830eed1afa324fc0e18b6e9f12cc8ee5a95e2d4e64c1215f54 Nov 01 20:30:07.120204 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.120179 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb754e3320d8dad235549d1c4b2afdd3.slice/crio-5da22848387f8defe679301e3673a5e493b731195038b4002d92af918b2eb6f0 WatchSource:0}: Error finding container 5da22848387f8defe679301e3673a5e493b731195038b4002d92af918b2eb6f0: Status 404 returned error can't find the container with id 5da22848387f8defe679301e3673a5e493b731195038b4002d92af918b2eb6f0 Nov 01 20:30:07.124542 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.124527 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:30:07.410097 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.410010 2578 apiserver.go:52] "Watching apiserver" Nov 01 20:30:07.439218 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.439185 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 20:30:07.562455 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.562423 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:30:07.562735 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.562715 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-mtclz","openshift-network-diagnostics/network-check-target-sgmhx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k","openshift-cluster-node-tuning-operator/tuned-mqgcd","openshift-dns/node-resolver-2lphr","openshift-ovn-kubernetes/ovnkube-node-hrnnn","kube-system/konnectivity-agent-9msm2","kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-658ds","openshift-image-registry/node-ca-nrcnn","openshift-multus/multus-cq22n","openshift-network-operator/iptables-alerter-kdh6b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-zp6f6"] Nov 01 20:30:07.566383 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.566365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:07.568468 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.568451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.570553 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.570537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.572569 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.572547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.574631 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.574612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.577522 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.577494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.580400 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.580381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.580510 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.580466 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:07.582640 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.582625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.584778 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.584761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:07.584852 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.584832 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:07.587002 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.586981 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.589382 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.589362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:07.589458 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.589430 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:07.592158 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.592140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.594655 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.594621 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"4758d5c314bffde2671732f370e32950","Type":"ContainerStarted","Data":"810b2f22f38bcd830eed1afa324fc0e18b6e9f12cc8ee5a95e2d4e64c1215f54"} Nov 01 20:30:07.594757 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.594663 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"fb754e3320d8dad235549d1c4b2afdd3","Type":"ContainerStarted","Data":"5da22848387f8defe679301e3673a5e493b731195038b4002d92af918b2eb6f0"} Nov 01 20:30:07.618509 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.618485 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:30:07.618611 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.618541 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:30:07.618611 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.618540 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r4cnd\"" Nov 01 20:30:07.619968 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.619947 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h288r\"" Nov 01 20:30:07.620253 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.620224 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:30:07.620350 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.620264 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:30:07.620409 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.620373 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:30:07.620473 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.620459 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:30:07.640106 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.640057 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 20:30:07.651708 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cnibin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.651811 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-systemd-units\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.651811 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651738 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40a69377-a3ea-42ab-8a85-753770827b97-serviceca\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.651811 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9z69\" (UniqueName: \"kubernetes.io/projected/9d633766-ee45-4299-a89f-7db3cd012d43-kube-api-access-s9z69\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.651952 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovn-node-metrics-cert\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.651952 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6v8m\" (UniqueName: \"kubernetes.io/projected/bade2e15-5673-4f0e-9fa4-fec7d544507c-kube-api-access-h6v8m\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.651952 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-conf\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.651952 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-lib-modules\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.651952 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4220aacf-5c03-4983-904f-6b0d522cb8c6-host-slash\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.652136 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.652136 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.651999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:07.652136 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c481a350-2dc5-450b-b59c-2a37640d458e-agent-certs\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:07.652136 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-slash\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652136 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.652310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-tmp-dir\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.652310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-bin\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcjb6\" (UniqueName: \"kubernetes.io/projected/40a69377-a3ea-42ab-8a85-753770827b97-kube-api-access-tcjb6\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.652310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-sys\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.652310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cni-binary-copy\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.652310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-netd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852xh\" (UniqueName: \"kubernetes.io/projected/65662b67-e317-41c9-959e-24c40741d793-kube-api-access-852xh\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-etc-tuned\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-hosts-file\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-ovn\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652456 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-run\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a69377-a3ea-42ab-8a85-753770827b97-host\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-sys-fs\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-daemon-config\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.652568 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-systemd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-var-lib-kubelet\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4220aacf-5c03-4983-904f-6b0d522cb8c6-iptables-alerter-script\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64fm9\" (UniqueName: \"kubernetes.io/projected/4220aacf-5c03-4983-904f-6b0d522cb8c6-kube-api-access-64fm9\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-netns\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-var-lib-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-script-lib\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-host\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-multus\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-kubelet\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.652835 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-env-overrides\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-kubernetes\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-tmp\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-bin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-etc-kubernetes\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmx8q\" (UniqueName: \"kubernetes.io/projected/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-kube-api-access-bmx8q\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-registration-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.652987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-multus-certs\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-log-socket\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-modprobe-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-kubelet-config\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-socket-dir-parent\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-k8s-cni-cncf-io\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-hostroot\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653223 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-config\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-systemd\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653276 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-etc-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-system-cni-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c481a350-2dc5-450b-b59c-2a37640d458e-konnectivity-ca\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-netns\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-socket-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-device-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-kubelet\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-cnibin\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-conf-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbhw\" (UniqueName: \"kubernetes.io/projected/f62b13d3-9a85-404d-b7c5-0a482d942bf3-kube-api-access-9tbhw\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysconfig\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.653693 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.654142 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-node-log\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.654142 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b27cw\" (UniqueName: \"kubernetes.io/projected/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kube-api-access-b27cw\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.654142 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-dbus\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.654142 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrpr\" (UniqueName: \"kubernetes.io/projected/1544a0e3-068d-4d70-bbfb-df3e4f86f391-kube-api-access-bhrpr\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:07.654142 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.653672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-system-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.714519 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.714432 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 20:30:07.714519 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.714495 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q5hz4" Nov 01 20:30:07.719335 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719312 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 20:30:07.719583 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719545 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:30:07.719583 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719572 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:30:07.719727 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719586 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9xjd\"" Nov 01 20:30:07.719727 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719576 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bdld\"" Nov 01 20:30:07.719807 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719755 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:30:07.719883 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719854 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:30:07.719883 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719875 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:07.720019 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719883 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:30:07.720019 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719898 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pjdf7\"" Nov 01 20:30:07.720019 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719920 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:30:07.720019 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719947 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:07.720019 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.719974 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720096 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720111 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720116 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720131 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ggf7k\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720119 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j72sw\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720159 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720163 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720161 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720227 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5cgzw\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720256 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720263 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720271 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:30:07.720328 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.720277 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wk8gn\"" Nov 01 20:30:07.729151 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.729128 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:30:07.729222 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.729188 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:30:07.754128 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-socket-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.754235 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-device-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.754235 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-kubelet\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.754235 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-cnibin\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.754235 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-conf-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.754235 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbhw\" (UniqueName: \"kubernetes.io/projected/f62b13d3-9a85-404d-b7c5-0a482d942bf3-kube-api-access-9tbhw\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-cnibin\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysconfig\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-conf-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-device-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-kubelet\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-node-log\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754313 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-socket-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysconfig\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27cw\" (UniqueName: \"kubernetes.io/projected/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kube-api-access-b27cw\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-node-log\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-dbus\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrpr\" (UniqueName: \"kubernetes.io/projected/1544a0e3-068d-4d70-bbfb-df3e4f86f391-kube-api-access-bhrpr\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:07.754432 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-system-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cnibin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-systemd-units\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-system-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40a69377-a3ea-42ab-8a85-753770827b97-serviceca\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-dbus\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9z69\" (UniqueName: \"kubernetes.io/projected/9d633766-ee45-4299-a89f-7db3cd012d43-kube-api-access-s9z69\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-systemd-units\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cnibin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovn-node-metrics-cert\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6v8m\" (UniqueName: \"kubernetes.io/projected/bade2e15-5673-4f0e-9fa4-fec7d544507c-kube-api-access-h6v8m\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-conf\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-lib-modules\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4220aacf-5c03-4983-904f-6b0d522cb8c6-host-slash\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754815 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4220aacf-5c03-4983-904f-6b0d522cb8c6-host-slash\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.755146 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c481a350-2dc5-450b-b59c-2a37640d458e-agent-certs\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-conf\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-slash\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-tmp-dir\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754970 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.754989 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-slash\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.754957 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755008 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-bin\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.755029 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcjb6\" (UniqueName: \"kubernetes.io/projected/40a69377-a3ea-42ab-8a85-753770827b97-kube-api-access-tcjb6\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755066 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.755095 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs podName:1544a0e3-068d-4d70-bbfb-df3e4f86f391 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:08.255055566 +0000 UTC m=+3.218062374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs") pod "network-metrics-daemon-mtclz" (UID: "1544a0e3-068d-4d70-bbfb-df3e4f86f391") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40a69377-a3ea-42ab-8a85-753770827b97-serviceca\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.755896 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-bin\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.755132 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret podName:689471e9-1161-4ae7-b08a-15b85ffaa359 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:08.255123771 +0000 UTC m=+3.218130551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret") pod "global-pull-secret-syncer-658ds" (UID: "689471e9-1161-4ae7-b08a-15b85ffaa359") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-sys\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cni-binary-copy\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-sys\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-netd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755251 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-tmp-dir\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755264 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-lib-modules\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755272 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-netd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852xh\" (UniqueName: \"kubernetes.io/projected/65662b67-e317-41c9-959e-24c40741d793-kube-api-access-852xh\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-etc-tuned\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-hosts-file\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-ovn\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-run\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a69377-a3ea-42ab-8a85-753770827b97-host\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-sys-fs\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.756860 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755482 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-daemon-config\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755501 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-hosts-file\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-systemd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755427 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-var-lib-kubelet\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755576 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-run\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a69377-a3ea-42ab-8a85-753770827b97-host\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4220aacf-5c03-4983-904f-6b0d522cb8c6-iptables-alerter-script\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-ovn\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755656 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-systemd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64fm9\" (UniqueName: \"kubernetes.io/projected/4220aacf-5c03-4983-904f-6b0d522cb8c6-kube-api-access-64fm9\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755708 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-sys-fs\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-var-lib-kubelet\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-netns\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755896 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-netns\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.757648 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-var-lib-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-script-lib\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755929 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-var-lib-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-host\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-multus\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-kubelet\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.755998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-host\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-env-overrides\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-kubernetes\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756048 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-multus\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-tmp\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-bin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-etc-kubernetes\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmx8q\" (UniqueName: \"kubernetes.io/projected/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-kube-api-access-bmx8q\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-kubernetes\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-registration-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-multus-certs\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-etc-kubernetes\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.758489 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-log-socket\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cni-binary-copy\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-modprobe-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-kubelet-config\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-socket-dir-parent\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-k8s-cni-cncf-io\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-hostroot\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-config\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-systemd\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756648 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-kubelet\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-etc-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756692 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-script-lib\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-system-cni-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.759286 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-systemd\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-system-cni-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c481a350-2dc5-450b-b59c-2a37640d458e-konnectivity-ca\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-netns\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-etc-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-modprobe-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.756872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757145 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-registration-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-multus-certs\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-env-overrides\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757246 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-log-socket\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-bin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-socket-dir-parent\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757437 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-netns\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757444 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-hostroot\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757494 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-kubelet-config\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:07.759819 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.757505 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-k8s-cni-cncf-io\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.758065 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c481a350-2dc5-450b-b59c-2a37640d458e-konnectivity-ca\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.758305 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.758920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-daemon-config\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.758986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-config\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.759015 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.759031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-etc-tuned\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.759145 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4220aacf-5c03-4983-904f-6b0d522cb8c6-iptables-alerter-script\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.759159 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.759175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovn-node-metrics-cert\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.759232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-tmp\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.760296 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.759437 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c481a350-2dc5-450b-b59c-2a37640d458e-agent-certs\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:07.811246 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.811216 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:07.811246 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.811247 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:07.811435 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.811262 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fqbjz for pod openshift-network-diagnostics/network-check-target-sgmhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:07.811435 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:07.811391 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz podName:96533abd-b0c9-46eb-bf51-96e4c58da84d nodeName:}" failed. No retries permitted until 2026-11-01 20:30:08.311326529 +0000 UTC m=+3.274333322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fqbjz" (UniqueName: "kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz") pod "network-check-target-sgmhx" (UID: "96533abd-b0c9-46eb-bf51-96e4c58da84d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:07.812788 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.812753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbhw\" (UniqueName: \"kubernetes.io/projected/f62b13d3-9a85-404d-b7c5-0a482d942bf3-kube-api-access-9tbhw\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.812898 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.812875 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrpr\" (UniqueName: \"kubernetes.io/projected/1544a0e3-068d-4d70-bbfb-df3e4f86f391-kube-api-access-bhrpr\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:07.812948 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.812925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27cw\" (UniqueName: \"kubernetes.io/projected/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kube-api-access-b27cw\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.812983 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.812943 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9z69\" (UniqueName: \"kubernetes.io/projected/9d633766-ee45-4299-a89f-7db3cd012d43-kube-api-access-s9z69\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.813623 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.813607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-852xh\" (UniqueName: \"kubernetes.io/projected/65662b67-e317-41c9-959e-24c40741d793-kube-api-access-852xh\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.813917 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.813894 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcjb6\" (UniqueName: \"kubernetes.io/projected/40a69377-a3ea-42ab-8a85-753770827b97-kube-api-access-tcjb6\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.814526 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.814508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6v8m\" (UniqueName: \"kubernetes.io/projected/bade2e15-5673-4f0e-9fa4-fec7d544507c-kube-api-access-h6v8m\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.814607 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.814580 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmx8q\" (UniqueName: \"kubernetes.io/projected/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-kube-api-access-bmx8q\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.814642 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.814602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64fm9\" (UniqueName: \"kubernetes.io/projected/4220aacf-5c03-4983-904f-6b0d522cb8c6-kube-api-access-64fm9\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.820932 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.820914 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:07.859705 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.859679 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:07.879846 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.879824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:07.886304 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.886283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cq22n" Nov 01 20:30:07.890490 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.890467 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc481a350_2dc5_450b_b59c_2a37640d458e.slice/crio-74d487b42f799296b93df915abf0f87624765ee4b7b204d57fb30d050f7fcc62 WatchSource:0}: Error finding container 74d487b42f799296b93df915abf0f87624765ee4b7b204d57fb30d050f7fcc62: Status 404 returned error can't find the container with id 74d487b42f799296b93df915abf0f87624765ee4b7b204d57fb30d050f7fcc62 Nov 01 20:30:07.896012 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.895988 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62b13d3_9a85_404d_b7c5_0a482d942bf3.slice/crio-1e6134c20cfa5a513930172e66ad149d003de882e6fc3b296f3c66921455c6df WatchSource:0}: Error finding container 1e6134c20cfa5a513930172e66ad149d003de882e6fc3b296f3c66921455c6df: Status 404 returned error can't find the container with id 1e6134c20cfa5a513930172e66ad149d003de882e6fc3b296f3c66921455c6df Nov 01 20:30:07.896584 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.896567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:07.897425 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.897409 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:07.902388 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.902305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:07.906145 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.906124 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d566de_6bf8_4fc4_966b_ac984e4a300e.slice/crio-f5b6bcb9c3083dd890446d460731ac6860c3ef5c51f106e1bf40c0c03cf728c6 WatchSource:0}: Error finding container f5b6bcb9c3083dd890446d460731ac6860c3ef5c51f106e1bf40c0c03cf728c6: Status 404 returned error can't find the container with id f5b6bcb9c3083dd890446d460731ac6860c3ef5c51f106e1bf40c0c03cf728c6 Nov 01 20:30:07.909550 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.909526 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d633766_ee45_4299_a89f_7db3cd012d43.slice/crio-24dc3d14aab40d3ff62677d1c6a13b933b4052988b7f5e408a7dd935e69319ce WatchSource:0}: Error finding container 24dc3d14aab40d3ff62677d1c6a13b933b4052988b7f5e408a7dd935e69319ce: Status 404 returned error can't find the container with id 24dc3d14aab40d3ff62677d1c6a13b933b4052988b7f5e408a7dd935e69319ce Nov 01 20:30:07.912405 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.912383 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e69a8f9_67ff_4ee6_9718_ab2c186021a1.slice/crio-0d2cd6301238788543f4b75f09925b0bf8fb65a15d55322b2effde78fc6ff453 WatchSource:0}: Error finding container 0d2cd6301238788543f4b75f09925b0bf8fb65a15d55322b2effde78fc6ff453: Status 404 returned error can't find the container with id 0d2cd6301238788543f4b75f09925b0bf8fb65a15d55322b2effde78fc6ff453 Nov 01 20:30:07.924585 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.924568 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:07.930425 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.930394 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:07.934042 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.934015 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a69377_a3ea_42ab_8a85_753770827b97.slice/crio-d0a5163eb731d22d1216988b4986c332b4d5c3847c664f32a387e88c72b3b0e8 WatchSource:0}: Error finding container d0a5163eb731d22d1216988b4986c332b4d5c3847c664f32a387e88c72b3b0e8: Status 404 returned error can't find the container with id d0a5163eb731d22d1216988b4986c332b4d5c3847c664f32a387e88c72b3b0e8 Nov 01 20:30:07.936177 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.936152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:07.938378 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.938355 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4220aacf_5c03_4983_904f_6b0d522cb8c6.slice/crio-dfdf69bf90e4c3b326ae94c316fb0097303a39e91cb5cfc5a4ed3507299fcd64 WatchSource:0}: Error finding container dfdf69bf90e4c3b326ae94c316fb0097303a39e91cb5cfc5a4ed3507299fcd64: Status 404 returned error can't find the container with id dfdf69bf90e4c3b326ae94c316fb0097303a39e91cb5cfc5a4ed3507299fcd64 Nov 01 20:30:07.942466 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:07.942450 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:07.944603 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.944584 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbade2e15_5673_4f0e_9fa4_fec7d544507c.slice/crio-30d306f736cc28a60a85486a7c9aae96c3190dc0a66e4901aa6a40a35ec2520f WatchSource:0}: Error finding container 30d306f736cc28a60a85486a7c9aae96c3190dc0a66e4901aa6a40a35ec2520f: Status 404 returned error can't find the container with id 30d306f736cc28a60a85486a7c9aae96c3190dc0a66e4901aa6a40a35ec2520f Nov 01 20:30:07.952041 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:07.952021 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65662b67_e317_41c9_959e_24c40741d793.slice/crio-41e122b4e06cd1eb7e68cffdab9f77bf7fd96ed9ad187f980dbd478b60734dde WatchSource:0}: Error finding container 41e122b4e06cd1eb7e68cffdab9f77bf7fd96ed9ad187f980dbd478b60734dde: Status 404 returned error can't find the container with id 41e122b4e06cd1eb7e68cffdab9f77bf7fd96ed9ad187f980dbd478b60734dde Nov 01 20:30:08.262899 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.262860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:08.263047 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.262913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:08.263156 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:08.263055 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:08.263156 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:08.263125 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret podName:689471e9-1161-4ae7-b08a-15b85ffaa359 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:09.263108562 +0000 UTC m=+4.226115345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret") pod "global-pull-secret-syncer-658ds" (UID: "689471e9-1161-4ae7-b08a-15b85ffaa359") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:08.263261 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:08.263187 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:08.263261 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:08.263216 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs podName:1544a0e3-068d-4d70-bbfb-df3e4f86f391 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:09.263205976 +0000 UTC m=+4.226212760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs") pod "network-metrics-daemon-mtclz" (UID: "1544a0e3-068d-4d70-bbfb-df3e4f86f391") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:08.363694 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.363642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:08.363846 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:08.363780 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:08.363846 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:08.363796 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:08.363846 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:08.363805 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fqbjz for pod openshift-network-diagnostics/network-check-target-sgmhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:08.363846 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:08.363842 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz podName:96533abd-b0c9-46eb-bf51-96e4c58da84d nodeName:}" failed. No retries permitted until 2026-11-01 20:30:09.363830368 +0000 UTC m=+4.326837147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqbjz" (UniqueName: "kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz") pod "network-check-target-sgmhx" (UID: "96533abd-b0c9-46eb-bf51-96e4c58da84d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:08.584445 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.584403 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerStarted","Data":"41e122b4e06cd1eb7e68cffdab9f77bf7fd96ed9ad187f980dbd478b60734dde"} Nov 01 20:30:08.587955 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.587412 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"30d306f736cc28a60a85486a7c9aae96c3190dc0a66e4901aa6a40a35ec2520f"} Nov 01 20:30:08.589337 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.589309 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdh6b" event={"ID":"4220aacf-5c03-4983-904f-6b0d522cb8c6","Type":"ContainerStarted","Data":"dfdf69bf90e4c3b326ae94c316fb0097303a39e91cb5cfc5a4ed3507299fcd64"} Nov 01 20:30:08.591161 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.591100 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9msm2" event={"ID":"c481a350-2dc5-450b-b59c-2a37640d458e","Type":"ContainerStarted","Data":"74d487b42f799296b93df915abf0f87624765ee4b7b204d57fb30d050f7fcc62"} Nov 01 20:30:08.592381 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.592361 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrcnn" event={"ID":"40a69377-a3ea-42ab-8a85-753770827b97","Type":"ContainerStarted","Data":"d0a5163eb731d22d1216988b4986c332b4d5c3847c664f32a387e88c72b3b0e8"} Nov 01 20:30:08.593698 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.593675 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2lphr" event={"ID":"2e69a8f9-67ff-4ee6-9718-ab2c186021a1","Type":"ContainerStarted","Data":"0d2cd6301238788543f4b75f09925b0bf8fb65a15d55322b2effde78fc6ff453"} Nov 01 20:30:08.595062 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.595013 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" event={"ID":"9d633766-ee45-4299-a89f-7db3cd012d43","Type":"ContainerStarted","Data":"24dc3d14aab40d3ff62677d1c6a13b933b4052988b7f5e408a7dd935e69319ce"} Nov 01 20:30:08.596403 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.596372 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" event={"ID":"e9d566de-6bf8-4fc4-966b-ac984e4a300e","Type":"ContainerStarted","Data":"f5b6bcb9c3083dd890446d460731ac6860c3ef5c51f106e1bf40c0c03cf728c6"} Nov 01 20:30:08.597715 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.597692 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cq22n" event={"ID":"f62b13d3-9a85-404d-b7c5-0a482d942bf3","Type":"ContainerStarted","Data":"1e6134c20cfa5a513930172e66ad149d003de882e6fc3b296f3c66921455c6df"} Nov 01 20:30:08.718203 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.718132 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:07 +0000 UTC" deadline="2028-05-14 19:14:02.284295531 +0000 UTC" Nov 01 20:30:08.718203 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:08.718196 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13438h43m53.566104037s" Nov 01 20:30:09.270959 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:09.270920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:09.271126 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:09.270973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:09.271188 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.271136 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:09.271236 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.271197 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret podName:689471e9-1161-4ae7-b08a-15b85ffaa359 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.271180182 +0000 UTC m=+6.234186968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret") pod "global-pull-secret-syncer-658ds" (UID: "689471e9-1161-4ae7-b08a-15b85ffaa359") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:09.271310 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.271257 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:09.271310 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.271289 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs podName:1544a0e3-068d-4d70-bbfb-df3e4f86f391 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.271277416 +0000 UTC m=+6.234284205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs") pod "network-metrics-daemon-mtclz" (UID: "1544a0e3-068d-4d70-bbfb-df3e4f86f391") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:09.371888 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:09.371806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:09.372040 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.371977 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:09.372040 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.371998 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:09.372040 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.372011 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fqbjz for pod openshift-network-diagnostics/network-check-target-sgmhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:09.372217 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.372065 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz podName:96533abd-b0c9-46eb-bf51-96e4c58da84d nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.372048134 +0000 UTC m=+6.335054930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqbjz" (UniqueName: "kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz") pod "network-check-target-sgmhx" (UID: "96533abd-b0c9-46eb-bf51-96e4c58da84d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:09.574651 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:09.574617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:09.574827 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.574761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:09.574912 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:09.574897 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:09.575030 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.575003 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:09.575163 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:09.575146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:09.575245 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:09.575226 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:09.720170 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:09.720040 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:07 +0000 UTC" deadline="2028-06-12 18:52:12.388778004 +0000 UTC" Nov 01 20:30:09.720170 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:09.720097 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14134h22m2.66868615s" Nov 01 20:30:10.613890 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:10.612701 2578 generic.go:356] "Generic (PLEG): container finished" podID="4758d5c314bffde2671732f370e32950" containerID="967e1550fed74dae51d22fcbd074ae693f8957537b662581cecf8ea88c0bad9e" exitCode=0 Nov 01 20:30:10.613890 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:10.613008 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"4758d5c314bffde2671732f370e32950","Type":"ContainerDied","Data":"967e1550fed74dae51d22fcbd074ae693f8957537b662581cecf8ea88c0bad9e"} Nov 01 20:30:10.624113 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:10.624061 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"fb754e3320d8dad235549d1c4b2afdd3","Type":"ContainerStarted","Data":"407a169d6b4af653460557d538b6122ed4f567ca9b71bd01558e1fc444075ea2"} Nov 01 20:30:11.287943 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.287915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:11.288420 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.287962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:11.288420 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.288115 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:11.288420 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.288167 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret podName:689471e9-1161-4ae7-b08a-15b85ffaa359 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:15.288149741 +0000 UTC m=+10.251156534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret") pod "global-pull-secret-syncer-658ds" (UID: "689471e9-1161-4ae7-b08a-15b85ffaa359") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:11.288420 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.288230 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:11.288420 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.288260 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs podName:1544a0e3-068d-4d70-bbfb-df3e4f86f391 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:15.288249414 +0000 UTC m=+10.251256195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs") pod "network-metrics-daemon-mtclz" (UID: "1544a0e3-068d-4d70-bbfb-df3e4f86f391") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:11.389673 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.389415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:11.389673 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.389559 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:11.389673 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.389576 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:11.389673 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.389588 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fqbjz for pod openshift-network-diagnostics/network-check-target-sgmhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:11.389673 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.389638 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz podName:96533abd-b0c9-46eb-bf51-96e4c58da84d nodeName:}" failed. No retries permitted until 2026-11-01 20:30:15.389622027 +0000 UTC m=+10.352628808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqbjz" (UniqueName: "kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz") pod "network-check-target-sgmhx" (UID: "96533abd-b0c9-46eb-bf51-96e4c58da84d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:11.575994 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.575733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:11.575994 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.575869 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:11.576248 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.576225 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:11.576305 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.576291 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:11.576360 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.576336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:11.576412 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:11.576401 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:11.630149 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.630042 2578 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="6a60f1ee0718e31475ec4324fed468dbf3cd14b82523869cd4adb12898560830" exitCode=0 Nov 01 20:30:11.631717 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.631042 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerDied","Data":"6a60f1ee0718e31475ec4324fed468dbf3cd14b82523869cd4adb12898560830"} Nov 01 20:30:11.636940 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.636913 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"4758d5c314bffde2671732f370e32950","Type":"ContainerStarted","Data":"1109f1eadff317845457dd22230493412d53de0fa518074e18a416476f8d919f"} Nov 01 20:30:11.748711 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:11.748647 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" podStartSLOduration=5.748625581 podStartE2EDuration="5.748625581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:10.841808685 +0000 UTC m=+5.804815486" watchObservedRunningTime="2026-11-01 20:30:11.748625581 +0000 UTC m=+6.711632383" Nov 01 20:30:12.645453 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:12.645374 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9msm2" event={"ID":"c481a350-2dc5-450b-b59c-2a37640d458e","Type":"ContainerStarted","Data":"36f62d55034417f49d2f37a1f4af5299d07fa46828a71771a0083520d3ae1f78"} Nov 01 20:30:12.757929 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:12.757338 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9msm2" podStartSLOduration=3.352378837 podStartE2EDuration="7.757319903s" totalImagesPullingTime="4.404941066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:12.756730477 +0000 UTC m=+7.719737279" watchObservedRunningTime="2026-11-01 20:30:12.757319903 +0000 UTC m=+7.720326706" Nov 01 20:30:12.757929 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:12.757737 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" podStartSLOduration=6.75772741 podStartE2EDuration="6.75772741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:11.861628549 +0000 UTC m=+6.824635354" watchObservedRunningTime="2026-11-01 20:30:12.75772741 +0000 UTC m=+7.720734213" Nov 01 20:30:13.576355 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:13.576287 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:13.576526 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:13.576417 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:13.576865 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:13.576846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:13.576965 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:13.576943 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:13.577022 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:13.577000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:13.577121 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:13.577102 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:14.660592 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:14.660549 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrcnn" event={"ID":"40a69377-a3ea-42ab-8a85-753770827b97","Type":"ContainerStarted","Data":"f17a5c09fbc23bd8b21de8aef59a8f002269d4eafd88476aa37c5b4ec3971395"} Nov 01 20:30:14.664765 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:14.664732 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" event={"ID":"e9d566de-6bf8-4fc4-966b-ac984e4a300e","Type":"ContainerStarted","Data":"4ba849cda657c441d7c41f88ea1287e9bc963b31335a21181116667dcc1cc2be"} Nov 01 20:30:15.325207 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:15.325165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:15.325356 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:15.325227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:15.325419 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.325396 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:15.325479 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.325464 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret podName:689471e9-1161-4ae7-b08a-15b85ffaa359 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:23.32544423 +0000 UTC m=+18.288451032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret") pod "global-pull-secret-syncer-658ds" (UID: "689471e9-1161-4ae7-b08a-15b85ffaa359") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:15.326012 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.325990 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:15.326100 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.326059 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs podName:1544a0e3-068d-4d70-bbfb-df3e4f86f391 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:23.326041893 +0000 UTC m=+18.289048706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs") pod "network-metrics-daemon-mtclz" (UID: "1544a0e3-068d-4d70-bbfb-df3e4f86f391") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:15.425815 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:15.425769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:15.425996 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.425962 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:15.426056 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.425986 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:15.426056 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.426010 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fqbjz for pod openshift-network-diagnostics/network-check-target-sgmhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:15.426170 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.426098 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz podName:96533abd-b0c9-46eb-bf51-96e4c58da84d nodeName:}" failed. No retries permitted until 2026-11-01 20:30:23.426056796 +0000 UTC m=+18.389063594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqbjz" (UniqueName: "kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz") pod "network-check-target-sgmhx" (UID: "96533abd-b0c9-46eb-bf51-96e4c58da84d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:15.575123 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:15.575069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:15.575334 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.575223 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:15.575626 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:15.575604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:15.575734 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.575714 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:15.575934 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:15.575918 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:15.576149 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:15.576020 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:16.244970 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.244945 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 20:30:16.509026 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.508864 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:30:16.244968639Z","UUID":"2afa008f-04d8-4938-a228-4291ed79b57a","Handler":null,"Name":"","Endpoint":""} Nov 01 20:30:16.511147 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.510881 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 20:30:16.511147 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.510901 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 20:30:16.673940 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.673899 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2lphr" event={"ID":"2e69a8f9-67ff-4ee6-9718-ab2c186021a1","Type":"ContainerStarted","Data":"02d2748c0093b24c00b00dd5c0bec7d1694ce9307aaf912c48a03a4c16afe8e6"} Nov 01 20:30:16.676324 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.676289 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" event={"ID":"e9d566de-6bf8-4fc4-966b-ac984e4a300e","Type":"ContainerStarted","Data":"992a8c3cda6665c96839e4d40c078091ed81f65fd577107d569b3737985bc49a"} Nov 01 20:30:16.678354 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.678327 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cq22n" event={"ID":"f62b13d3-9a85-404d-b7c5-0a482d942bf3","Type":"ContainerStarted","Data":"f6e56261ae802ab797d1527340e1b964c35d161dfcae20d3fc6ee9fe197cbd88"} Nov 01 20:30:16.680890 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.680867 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerStarted","Data":"6f8fc19beb902db8a94c91c1facf2deb2ac00c255a3d3814fdf3d8c13169379f"} Nov 01 20:30:16.788193 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.788137 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nrcnn" podStartSLOduration=5.982897116 podStartE2EDuration="11.788119034s" totalImagesPullingTime="5.805221918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:14.77276479 +0000 UTC m=+9.735771590" watchObservedRunningTime="2026-11-01 20:30:16.788119034 +0000 UTC m=+11.751125833" Nov 01 20:30:16.788651 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:16.788619 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2lphr" podStartSLOduration=3.359295764 podStartE2EDuration="11.788609522s" totalImagesPullingTime="8.429313758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:16.787919482 +0000 UTC m=+11.750926283" watchObservedRunningTime="2026-11-01 20:30:16.788609522 +0000 UTC m=+11.751616327" Nov 01 20:30:17.017309 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:17.017198 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cq22n" podStartSLOduration=3.988307433 podStartE2EDuration="12.017179667s" totalImagesPullingTime="8.028872234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:17.017142124 +0000 UTC m=+11.980148928" watchObservedRunningTime="2026-11-01 20:30:17.017179667 +0000 UTC m=+11.980186469" Nov 01 20:30:17.561314 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:17.561279 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:17.562149 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:17.562127 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:17.574274 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:17.574243 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:17.574442 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:17.574384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:17.574442 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:17.574403 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:17.574509 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:17.574485 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:17.574548 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:17.574536 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:17.574613 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:17.574594 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:17.684873 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:17.684838 2578 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="6f8fc19beb902db8a94c91c1facf2deb2ac00c255a3d3814fdf3d8c13169379f" exitCode=0 Nov 01 20:30:17.685031 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:17.684958 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerDied","Data":"6f8fc19beb902db8a94c91c1facf2deb2ac00c255a3d3814fdf3d8c13169379f"} Nov 01 20:30:18.696903 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:18.696658 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdh6b" event={"ID":"4220aacf-5c03-4983-904f-6b0d522cb8c6","Type":"ContainerStarted","Data":"d242d4e0066535b9933c3796cffb79274f2a10c3c06658e1f09fadff0c55c571"} Nov 01 20:30:18.698546 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:18.698512 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" event={"ID":"9d633766-ee45-4299-a89f-7db3cd012d43","Type":"ContainerStarted","Data":"10ab364e23e80cb001b1b3b5f8bbf340e182699a30b6ea1a15e0782d4bf2038a"} Nov 01 20:30:18.701053 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:18.701027 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" event={"ID":"e9d566de-6bf8-4fc4-966b-ac984e4a300e","Type":"ContainerStarted","Data":"88ab504f91759343136b7619e8d47aa0816b39aaee4fea20f90f6c9802bfe3db"} Nov 01 20:30:18.810897 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:18.810774 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kdh6b" podStartSLOduration=5.263645699 podStartE2EDuration="13.810754125s" totalImagesPullingTime="8.547108426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:18.81044368 +0000 UTC m=+13.773450480" watchObservedRunningTime="2026-11-01 20:30:18.810754125 +0000 UTC m=+13.773760927" Nov 01 20:30:18.924488 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:18.924428 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" podStartSLOduration=4.243017106 podStartE2EDuration="13.924409687s" totalImagesPullingTime="9.681392581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:18.924353101 +0000 UTC m=+13.887359931" watchObservedRunningTime="2026-11-01 20:30:18.924409687 +0000 UTC m=+13.887416482" Nov 01 20:30:19.037380 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:19.037332 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" podStartSLOduration=3.415262747 podStartE2EDuration="14.03731333s" totalImagesPullingTime="10.622050583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:19.037179022 +0000 UTC m=+14.000185824" watchObservedRunningTime="2026-11-01 20:30:19.03731333 +0000 UTC m=+14.000320131" Nov 01 20:30:19.574490 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:19.574456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:19.574490 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:19.574484 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:19.574730 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:19.574524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:19.574730 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:19.574604 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:19.574730 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:19.574671 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:19.574876 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:19.574765 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:19.705676 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:19.705634 2578 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="095ed284e981ac674fad2faf275e548098f850bb742af423903cafc6947865a0" exitCode=0 Nov 01 20:30:19.706406 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:19.705762 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerDied","Data":"095ed284e981ac674fad2faf275e548098f850bb742af423903cafc6947865a0"} Nov 01 20:30:21.574148 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:21.574105 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:21.574791 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:21.574166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:21.574791 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:21.574172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:21.574791 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:21.574267 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:21.574791 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:21.574423 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:21.574791 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:21.574601 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:21.712682 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:21.712646 2578 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="233bbfb1b884b2175d01b7ce1fe61e3e02e3cc06aa659df76129c0d561de03ea" exitCode=0 Nov 01 20:30:21.712849 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:21.712700 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerDied","Data":"233bbfb1b884b2175d01b7ce1fe61e3e02e3cc06aa659df76129c0d561de03ea"} Nov 01 20:30:23.391487 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.391307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:23.391865 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.391502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:23.391865 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.391456 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:23.391865 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.391628 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:23.391865 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.391638 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs podName:1544a0e3-068d-4d70-bbfb-df3e4f86f391 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.391615174 +0000 UTC m=+34.354621955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs") pod "network-metrics-daemon-mtclz" (UID: "1544a0e3-068d-4d70-bbfb-df3e4f86f391") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:23.391865 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.391676 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret podName:689471e9-1161-4ae7-b08a-15b85ffaa359 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.391662634 +0000 UTC m=+34.354669413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret") pod "global-pull-secret-syncer-658ds" (UID: "689471e9-1161-4ae7-b08a-15b85ffaa359") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:23.492150 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.492120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:23.492283 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.492266 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:23.492332 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.492287 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:23.492332 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.492301 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fqbjz for pod openshift-network-diagnostics/network-check-target-sgmhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:23.492404 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.492352 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz podName:96533abd-b0c9-46eb-bf51-96e4c58da84d nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.492335549 +0000 UTC m=+34.455342346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqbjz" (UniqueName: "kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz") pod "network-check-target-sgmhx" (UID: "96533abd-b0c9-46eb-bf51-96e4c58da84d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:23.574003 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.573967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:23.574164 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.574114 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:23.574885 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.574864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:23.574975 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.574959 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:23.575026 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.575016 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:23.575105 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:23.575087 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:23.720037 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.719995 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"260ccfc72e81eb50f7b37eefeff068c0f0922347e87c0140bef25c2837e769d3"} Nov 01 20:30:23.720213 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.720064 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"5a71464ed563f1d25ee932c1f8e391f809c4003b46909bcafd0de459eddbe9f7"} Nov 01 20:30:23.720213 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.720118 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"23db449f86395925c73ff65ad33eb9284dd40c194d4cd05bffcc3009e5c30517"} Nov 01 20:30:23.720213 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.720151 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"ab5d7b061e0c79631780e8171ae09b51a0afa256a18a5c6395698c49e116ec07"} Nov 01 20:30:23.720213 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.720177 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"b15727ec003779a2f28d43b3c20b76cc8f66e2db5b2ddb0e1e78a9eeacf5f460"} Nov 01 20:30:23.720213 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:23.720202 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"2794f7a4d0595c67bb9b39629f0047e6143700b259579a65cfa953c1e65d27de"} Nov 01 20:30:24.724554 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:24.724301 2578 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="b367291c4c1f005cd4ac8a768ba2edb97d05d5cb45c23680a97723fa751d7af2" exitCode=0 Nov 01 20:30:24.724974 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:24.724384 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerDied","Data":"b367291c4c1f005cd4ac8a768ba2edb97d05d5cb45c23680a97723fa751d7af2"} Nov 01 20:30:24.951676 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:24.951645 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:24.951885 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:24.951802 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:30:24.952300 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:24.952279 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:25.574793 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:25.574761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:25.574961 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:25.574841 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:25.574961 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:25.574887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:25.574961 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:25.574932 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:25.574961 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:25.574959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:25.575113 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:25.575001 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:25.730118 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:25.730058 2578 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="ceb8eb59e36a638da26c1588587b643f73107c67655864431c1aa7aee870b7a9" exitCode=0 Nov 01 20:30:25.730118 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:25.730111 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerDied","Data":"ceb8eb59e36a638da26c1588587b643f73107c67655864431c1aa7aee870b7a9"} Nov 01 20:30:26.735645 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:26.735609 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" event={"ID":"65662b67-e317-41c9-959e-24c40741d793","Type":"ContainerStarted","Data":"5eb1fa814cb4f3cc94d77fe6bbeb0d3638fadb5f85ecd688a9e2504b68cd109d"} Nov 01 20:30:26.738847 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:26.738821 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"520b2483e13be1b96f830178a690976de188ad74d29620973280cdc65d8cc8a4"} Nov 01 20:30:26.851616 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:26.851561 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zp6f6" podStartSLOduration=8.921619442 podStartE2EDuration="21.851542953s" totalImagesPullingTime="12.929923511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:26.851121731 +0000 UTC m=+21.814128529" watchObservedRunningTime="2026-11-01 20:30:26.851542953 +0000 UTC m=+21.814549756" Nov 01 20:30:27.573523 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:27.573486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:27.573523 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:27.573509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:27.573869 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:27.573499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:27.573869 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:27.573586 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:27.573869 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:27.573651 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:27.573869 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:27.573756 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:28.745784 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:28.745751 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" event={"ID":"bade2e15-5673-4f0e-9fa4-fec7d544507c","Type":"ContainerStarted","Data":"ccaa11d936b46595cb81ecc2ffe648b98db6ce347d896f73ebd1b75ad42ea968"} Nov 01 20:30:28.746524 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:28.746090 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:28.762273 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:28.762248 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:29.014509 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:29.014374 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" podStartSLOduration=8.66614205 podStartE2EDuration="24.014354314s" totalImagesPullingTime="15.348212264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:28.962269678 +0000 UTC m=+23.925276480" watchObservedRunningTime="2026-11-01 20:30:29.014354314 +0000 UTC m=+23.977361119" Nov 01 20:30:29.574035 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:29.573999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:29.574240 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:29.574157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:29.574240 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:29.574169 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:29.574352 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:29.574161 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:29.574352 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:29.574261 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:29.574444 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:29.574344 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:29.748792 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:29.748762 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:29.749228 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:29.748842 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:29.764149 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:29.764118 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:30.582401 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:30.582225 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-658ds"] Nov 01 20:30:30.582602 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:30.582481 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:30.582602 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:30.582577 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:30.585218 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:30.585190 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mtclz"] Nov 01 20:30:30.585347 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:30.585297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:30.585409 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:30.585395 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:30.585703 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:30.585681 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sgmhx"] Nov 01 20:30:30.585793 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:30.585759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:30.585863 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:30.585836 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:32.574052 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:32.574025 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:32.574397 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:32.574052 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:32.574397 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:32.574164 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:32.574397 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:32.574216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:32.574397 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:32.574334 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:32.574598 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:32.574435 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:34.573886 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:34.573842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:34.573886 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:34.573870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:34.573886 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:34.573885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:34.574540 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:34.573974 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgmhx" podUID="96533abd-b0c9-46eb-bf51-96e4c58da84d" Nov 01 20:30:34.574540 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:34.574029 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-658ds" podUID="689471e9-1161-4ae7-b08a-15b85ffaa359" Nov 01 20:30:34.574540 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:34.574128 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtclz" podUID="1544a0e3-068d-4d70-bbfb-df3e4f86f391" Nov 01 20:30:35.824923 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:35.824893 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeReady" Nov 01 20:30:35.825310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:35.825014 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 20:30:35.976809 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:35.976769 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w9xfp"] Nov 01 20:30:35.978768 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:35.978748 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bkh8x"] Nov 01 20:30:35.978891 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:35.978865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:35.980495 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:35.980473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:36.030818 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.030787 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:30:36.030818 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.030801 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:30:36.031032 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.030833 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n2j5c\"" Nov 01 20:30:36.031760 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.031745 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:30:36.032008 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.031991 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:30:36.032117 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.032045 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:30:36.032172 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.032126 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jgj4\"" Nov 01 20:30:36.050085 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.050044 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bkh8x"] Nov 01 20:30:36.051777 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.051756 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w9xfp"] Nov 01 20:30:36.083325 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.083294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.083325 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.083332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9a1fb0-d057-4922-97a5-62ea9bebf833-tmp-dir\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.083516 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.083378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mqs\" (UniqueName: \"kubernetes.io/projected/6d9a1fb0-d057-4922-97a5-62ea9bebf833-kube-api-access-j5mqs\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.083516 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.083484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9h84\" (UniqueName: \"kubernetes.io/projected/04878621-4728-4a68-8e75-1af069bec931-kube-api-access-m9h84\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:36.083575 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.083559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1fb0-d057-4922-97a5-62ea9bebf833-config-volume\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.083611 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.083587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04878621-4728-4a68-8e75-1af069bec931-cert\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:36.184236 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.184155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9a1fb0-d057-4922-97a5-62ea9bebf833-tmp-dir\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.184236 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.184208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mqs\" (UniqueName: \"kubernetes.io/projected/6d9a1fb0-d057-4922-97a5-62ea9bebf833-kube-api-access-j5mqs\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.184236 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.184228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9h84\" (UniqueName: \"kubernetes.io/projected/04878621-4728-4a68-8e75-1af069bec931-kube-api-access-m9h84\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:36.184499 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.184261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1fb0-d057-4922-97a5-62ea9bebf833-config-volume\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.184499 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.184287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04878621-4728-4a68-8e75-1af069bec931-cert\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:36.184499 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.184308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.184499 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:36.184395 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:36.184499 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:36.184452 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls podName:6d9a1fb0-d057-4922-97a5-62ea9bebf833 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:36.684439229 +0000 UTC m=+31.647446021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls") pod "dns-default-w9xfp" (UID: "6d9a1fb0-d057-4922-97a5-62ea9bebf833") : secret "dns-default-metrics-tls" not found Nov 01 20:30:36.184787 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.184591 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9a1fb0-d057-4922-97a5-62ea9bebf833-tmp-dir\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.184889 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.184836 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1fb0-d057-4922-97a5-62ea9bebf833-config-volume\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.188228 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.188204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04878621-4728-4a68-8e75-1af069bec931-cert\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:36.243685 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.243650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mqs\" (UniqueName: \"kubernetes.io/projected/6d9a1fb0-d057-4922-97a5-62ea9bebf833-kube-api-access-j5mqs\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.243820 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.243735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9h84\" (UniqueName: \"kubernetes.io/projected/04878621-4728-4a68-8e75-1af069bec931-kube-api-access-m9h84\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:36.295092 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.295042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:36.573642 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.573613 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:36.573776 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.573692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:36.573847 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.573799 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:36.626034 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.626010 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:30:36.626277 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.626258 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:30:36.626378 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.626295 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:30:36.626378 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.626364 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kzbx4\"" Nov 01 20:30:36.626479 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.626451 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cg54x\"" Nov 01 20:30:36.626524 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.626478 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:30:36.626524 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.626492 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pskvw\"" Nov 01 20:30:36.680242 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.680216 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bkh8x"] Nov 01 20:30:36.683554 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:36.683530 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04878621_4728_4a68_8e75_1af069bec931.slice/crio-9dc3a90a021157de8a3abf5474b5abecaf830511ba7bfbbb14c9bcd118150aa4 WatchSource:0}: Error finding container 9dc3a90a021157de8a3abf5474b5abecaf830511ba7bfbbb14c9bcd118150aa4: Status 404 returned error can't find the container with id 9dc3a90a021157de8a3abf5474b5abecaf830511ba7bfbbb14c9bcd118150aa4 Nov 01 20:30:36.687213 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.687192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:36.687337 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:36.687323 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:36.687397 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:36.687386 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls podName:6d9a1fb0-d057-4922-97a5-62ea9bebf833 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:37.687370839 +0000 UTC m=+32.650377635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls") pod "dns-default-w9xfp" (UID: "6d9a1fb0-d057-4922-97a5-62ea9bebf833") : secret "dns-default-metrics-tls" not found Nov 01 20:30:36.767116 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:36.767063 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bkh8x" event={"ID":"04878621-4728-4a68-8e75-1af069bec931","Type":"ContainerStarted","Data":"9dc3a90a021157de8a3abf5474b5abecaf830511ba7bfbbb14c9bcd118150aa4"} Nov 01 20:30:37.693642 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:37.693604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:37.694023 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:37.693792 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:37.694023 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:37.693878 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls podName:6d9a1fb0-d057-4922-97a5-62ea9bebf833 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.693857862 +0000 UTC m=+34.656864646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls") pod "dns-default-w9xfp" (UID: "6d9a1fb0-d057-4922-97a5-62ea9bebf833") : secret "dns-default-metrics-tls" not found Nov 01 20:30:38.433482 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.433446 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xtldx"] Nov 01 20:30:38.450901 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.450833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.502673 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.502648 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:30:38.502673 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.502646 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:30:38.502904 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.502855 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:30:38.552546 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.552503 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:30:38.552683 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.552558 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t8z8c\"" Nov 01 20:30:38.553829 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.553801 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xtldx"] Nov 01 20:30:38.600041 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.600012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlt4b\" (UniqueName: \"kubernetes.io/projected/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-api-access-nlt4b\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.600194 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.600061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ace2d1f7-e48a-4375-848d-4cd0f9122046-data-volume\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.600257 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.600212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.600310 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.600263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.600373 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.600353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ace2d1f7-e48a-4375-848d-4cd0f9122046-crio-socket\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.682805 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.682770 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8mzmm-bpj4x"] Nov 01 20:30:38.701273 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.701192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ace2d1f7-e48a-4375-848d-4cd0f9122046-data-volume\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.701273 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.701272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.701326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.701354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ace2d1f7-e48a-4375-848d-4cd0f9122046-crio-socket\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.701385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlt4b\" (UniqueName: \"kubernetes.io/projected/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-api-access-nlt4b\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:38.701493 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:38.701584 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.201561635 +0000 UTC m=+34.164568431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : configmap "kube-rbac-proxy" not found Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.701598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ace2d1f7-e48a-4375-848d-4cd0f9122046-data-volume\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:38.701707 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:38.701780 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.201761993 +0000 UTC m=+34.164768792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:38.701803 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.701802 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ace2d1f7-e48a-4375-848d-4cd0f9122046-crio-socket\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.703030 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.703012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.741878 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.741846 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8mzmm-bpj4x"] Nov 01 20:30:38.755762 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.755726 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 20:30:38.755983 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.755964 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9m2k9\"" Nov 01 20:30:38.801994 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.801954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.801994 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.801991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxvs\" (UniqueName: \"kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.802265 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.802020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.802265 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.802246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.855287 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.855245 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 20:30:38.857958 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.857933 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlt4b\" (UniqueName: \"kubernetes.io/projected/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-api-access-nlt4b\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:38.902759 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.902719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.902922 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.902776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.902922 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.902873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxvs\" (UniqueName: \"kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.902922 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.902915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.903114 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.903100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.903379 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.903363 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.903654 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.903637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:38.960108 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:38.960014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxvs\" (UniqueName: \"kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:39.035811 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.035778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:39.206213 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.206163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:39.206367 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.206257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:39.206367 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:39.206306 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 20:30:39.206367 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:39.206360 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:39.206518 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:39.206396 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:40.206372836 +0000 UTC m=+35.169379620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : configmap "kube-rbac-proxy" not found Nov 01 20:30:39.206518 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:39.206428 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:40.206406394 +0000 UTC m=+35.169413177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:39.407999 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.407964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:39.407999 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.408003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:39.408224 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:39.408146 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 20:30:39.408224 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:39.408216 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs podName:1544a0e3-068d-4d70-bbfb-df3e4f86f391 nodeName:}" failed. No retries permitted until 2026-11-01 20:31:11.408201265 +0000 UTC m=+66.371208067 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs") pod "network-metrics-daemon-mtclz" (UID: "1544a0e3-068d-4d70-bbfb-df3e4f86f391") : secret "metrics-daemon-secret" not found Nov 01 20:30:39.410673 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.410647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:39.435982 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.435941 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8mzmm-bpj4x"] Nov 01 20:30:39.440754 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:39.440729 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d29d9e_1400_40e2_b20b_793790117269.slice/crio-876b58cf27baf26f9cac87ab83427cb7cec697d9ba95b5dd1c2abbca0d8fd775 WatchSource:0}: Error finding container 876b58cf27baf26f9cac87ab83427cb7cec697d9ba95b5dd1c2abbca0d8fd775: Status 404 returned error can't find the container with id 876b58cf27baf26f9cac87ab83427cb7cec697d9ba95b5dd1c2abbca0d8fd775 Nov 01 20:30:39.509175 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.509127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:39.512193 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.512168 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:39.652970 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.652934 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:39.673211 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.673128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:39.710396 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.710364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:39.710833 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:39.710525 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:39.710833 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:39.710577 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls podName:6d9a1fb0-d057-4922-97a5-62ea9bebf833 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:43.710563547 +0000 UTC m=+38.673570326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls") pod "dns-default-w9xfp" (UID: "6d9a1fb0-d057-4922-97a5-62ea9bebf833") : secret "dns-default-metrics-tls" not found Nov 01 20:30:39.777735 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:39.777710 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" event={"ID":"f6d29d9e-1400-40e2-b20b-793790117269","Type":"ContainerStarted","Data":"876b58cf27baf26f9cac87ab83427cb7cec697d9ba95b5dd1c2abbca0d8fd775"} Nov 01 20:30:40.060879 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:40.060712 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-658ds"] Nov 01 20:30:40.063329 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:40.063298 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689471e9_1161_4ae7_b08a_15b85ffaa359.slice/crio-1a90063b8a53234caa6c5762c060bc2452ed5b611c47e5e4247f570ed98d4e90 WatchSource:0}: Error finding container 1a90063b8a53234caa6c5762c060bc2452ed5b611c47e5e4247f570ed98d4e90: Status 404 returned error can't find the container with id 1a90063b8a53234caa6c5762c060bc2452ed5b611c47e5e4247f570ed98d4e90 Nov 01 20:30:40.109939 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:40.109894 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sgmhx"] Nov 01 20:30:40.113422 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:40.113389 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96533abd_b0c9_46eb_bf51_96e4c58da84d.slice/crio-cf055069953384ea52cab63f58530cb986ab941b7040455d75eed04e2d890608 WatchSource:0}: Error finding container cf055069953384ea52cab63f58530cb986ab941b7040455d75eed04e2d890608: Status 404 returned error can't find the container with id cf055069953384ea52cab63f58530cb986ab941b7040455d75eed04e2d890608 Nov 01 20:30:40.214664 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:40.214625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:40.214850 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:40.214693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:40.214850 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:40.214768 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 20:30:40.214850 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:40.214809 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:40.214850 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:40.214850 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:42.214830924 +0000 UTC m=+37.177837720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : configmap "kube-rbac-proxy" not found Nov 01 20:30:40.215054 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:40.214874 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:42.214860933 +0000 UTC m=+37.177867712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:40.781040 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:40.780953 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sgmhx" event={"ID":"96533abd-b0c9-46eb-bf51-96e4c58da84d","Type":"ContainerStarted","Data":"cf055069953384ea52cab63f58530cb986ab941b7040455d75eed04e2d890608"} Nov 01 20:30:40.782128 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:40.782106 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-658ds" event={"ID":"689471e9-1161-4ae7-b08a-15b85ffaa359","Type":"ContainerStarted","Data":"1a90063b8a53234caa6c5762c060bc2452ed5b611c47e5e4247f570ed98d4e90"} Nov 01 20:30:40.783372 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:40.783351 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bkh8x" event={"ID":"04878621-4728-4a68-8e75-1af069bec931","Type":"ContainerStarted","Data":"f6d442622b6debe340f256e070e7f605c45b1672906ebc22c98aea38c24101b1"} Nov 01 20:30:40.899521 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:40.899449 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bkh8x" podStartSLOduration=2.34101486 podStartE2EDuration="5.899428057s" totalImagesPullingTime="3.558413197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:40.898445771 +0000 UTC m=+35.861452578" watchObservedRunningTime="2026-11-01 20:30:40.899428057 +0000 UTC m=+35.862434859" Nov 01 20:30:41.049398 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.049303 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn"] Nov 01 20:30:41.064446 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.064412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" Nov 01 20:30:41.109093 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.109048 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn"] Nov 01 20:30:41.118247 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.118219 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:41.118406 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.118388 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4gdtb\"" Nov 01 20:30:41.118667 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.118628 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 20:30:41.224925 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.224890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dznxf\" (UniqueName: \"kubernetes.io/projected/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09-kube-api-access-dznxf\") pod \"migrator-66df5d496b-kshpn\" (UID: \"0c893f01-c22e-4e7d-a4ce-8e7896ef0e09\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" Nov 01 20:30:41.326192 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.326150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznxf\" (UniqueName: \"kubernetes.io/projected/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09-kube-api-access-dznxf\") pod \"migrator-66df5d496b-kshpn\" (UID: \"0c893f01-c22e-4e7d-a4ce-8e7896ef0e09\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" Nov 01 20:30:41.386130 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.386097 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznxf\" (UniqueName: \"kubernetes.io/projected/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09-kube-api-access-dznxf\") pod \"migrator-66df5d496b-kshpn\" (UID: \"0c893f01-c22e-4e7d-a4ce-8e7896ef0e09\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" Nov 01 20:30:41.388799 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.388777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" Nov 01 20:30:41.793989 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:41.791544 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn"] Nov 01 20:30:41.795708 ip-10-0-133-242 kubenswrapper[2578]: W1101 20:30:41.795682 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c893f01_c22e_4e7d_a4ce_8e7896ef0e09.slice/crio-ed4cb2e867b390c75069bf863ac23d8617aff6381efab7bb728c2e866890a20f WatchSource:0}: Error finding container ed4cb2e867b390c75069bf863ac23d8617aff6381efab7bb728c2e866890a20f: Status 404 returned error can't find the container with id ed4cb2e867b390c75069bf863ac23d8617aff6381efab7bb728c2e866890a20f Nov 01 20:30:42.234544 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:42.234505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:42.234728 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:42.234573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:42.234728 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:42.234672 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 20:30:42.234728 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:42.234680 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:42.234871 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:42.234745 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:46.234725118 +0000 UTC m=+41.197731922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:42.234871 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:42.234765 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:46.234755852 +0000 UTC m=+41.197762647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : configmap "kube-rbac-proxy" not found Nov 01 20:30:42.791262 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:42.791220 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" event={"ID":"0c893f01-c22e-4e7d-a4ce-8e7896ef0e09","Type":"ContainerStarted","Data":"ed4cb2e867b390c75069bf863ac23d8617aff6381efab7bb728c2e866890a20f"} Nov 01 20:30:43.748642 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:43.748345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:43.750148 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:43.749613 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:43.750148 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:43.749713 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls podName:6d9a1fb0-d057-4922-97a5-62ea9bebf833 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:51.749687086 +0000 UTC m=+46.712693875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls") pod "dns-default-w9xfp" (UID: "6d9a1fb0-d057-4922-97a5-62ea9bebf833") : secret "dns-default-metrics-tls" not found Nov 01 20:30:43.796577 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:43.796535 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" event={"ID":"f6d29d9e-1400-40e2-b20b-793790117269","Type":"ContainerStarted","Data":"57ff9e150e68bd39911730ae91dc728deae0fdf0e9123d203462b1d331f431ca"} Nov 01 20:30:43.909406 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:43.909339 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" podStartSLOduration=2.027628698 podStartE2EDuration="5.909317096s" totalImagesPullingTime="3.881688398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:43.908561029 +0000 UTC m=+38.871567830" watchObservedRunningTime="2026-11-01 20:30:43.909317096 +0000 UTC m=+38.872323899" Nov 01 20:30:44.800863 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:44.800810 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" event={"ID":"0c893f01-c22e-4e7d-a4ce-8e7896ef0e09","Type":"ContainerStarted","Data":"6e34fb74d74d8c5d9c89e30187739c33a2cf94374b2e9b5e1af356e5df41ff73"} Nov 01 20:30:44.802512 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:44.802479 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sgmhx" event={"ID":"96533abd-b0c9-46eb-bf51-96e4c58da84d","Type":"ContainerStarted","Data":"d6aa83a119716c1882d9c11a6e8d73f1d978d982009f7b591ba4e6195afe2f69"} Nov 01 20:30:44.914402 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:44.914336 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" podStartSLOduration=1.537823859 podStartE2EDuration="3.914314571s" totalImagesPullingTime="2.376490712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:44.913094248 +0000 UTC m=+39.876101054" watchObservedRunningTime="2026-11-01 20:30:44.914314571 +0000 UTC m=+39.877321373" Nov 01 20:30:45.026176 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:45.026041 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sgmhx" podStartSLOduration=35.450049398 podStartE2EDuration="40.026019491s" totalImagesPullingTime="4.575970093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:45.025035188 +0000 UTC m=+39.988041990" watchObservedRunningTime="2026-11-01 20:30:45.026019491 +0000 UTC m=+39.989026293" Nov 01 20:30:45.805430 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:45.805396 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:46.269458 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:46.269420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:46.269661 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:46.269473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:46.269661 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:46.269601 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:46.269779 ip-10-0-133-242 kubenswrapper[2578]: E1101 20:30:46.269685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:54.269664281 +0000 UTC m=+49.232671062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:46.269951 ip-10-0-133-242 kubenswrapper[2578]: I1101 20:30:46.269928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:46.275219 ip-10-0-133-242 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 20:30:46.293493 ip-10-0-133-242 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 20:30:46.293670 ip-10-0-133-242 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 20:30:46.293723 ip-10-0-133-242 systemd[1]: kubelet.service: Consumed 2.492s CPU time. Nov 01 20:30:46.308677 ip-10-0-133-242 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:30:46.424338 ip-10-0-133-242 kubenswrapper[4477]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:46.424338 ip-10-0-133-242 kubenswrapper[4477]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:30:46.424338 ip-10-0-133-242 kubenswrapper[4477]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:46.424338 ip-10-0-133-242 kubenswrapper[4477]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:46.426396 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426313 4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:46.426396 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426396 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:46.426396 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426400 4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426403 4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426406 4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426409 4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426411 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426414 4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426417 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426420 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426423 4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426426 4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426428 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426438 4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426449 4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426454 4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426456 4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426459 4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426462 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426464 4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426467 4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:46.426501 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426469 4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426472 4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426474 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426476 4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426479 4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426481 4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426484 4477 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426488 4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426492 4477 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426495 4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426498 4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426501 4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426504 4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426506 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426509 4477 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426512 4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426515 4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426518 4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426520 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426523 4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:46.427017 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426525 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426528 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426531 4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426534 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426536 4477 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426539 4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426542 4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426544 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426547 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426549 4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426552 4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426554 4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426557 4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426559 4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426562 4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426565 4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426568 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426570 4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426573 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:46.427539 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426575 4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426579 4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426583 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426586 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426589 4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426591 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426594 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426596 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426599 4477 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426601 4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426603 4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426606 4477 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426609 4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426611 4477 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426646 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426650 4477 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426653 4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426656 4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426659 4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426669 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:46.428016 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426671 4477 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426674 4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426676 4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426679 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426681 4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426684 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426687 4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426690 4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426693 4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426695 4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426698 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426700 4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426703 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426705 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426708 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426711 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426713 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426716 4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426718 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426721 4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:46.428547 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426723 4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426725 4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426728 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426730 4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426733 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.426735 4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427513 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427520 4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427523 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427526 4477 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427529 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427535 4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427538 4477 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427541 4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427544 4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427547 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427550 4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427553 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427556 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427558 4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:46.429041 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427561 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427563 4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427566 4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427568 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427571 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427573 4477 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427576 4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427579 4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427581 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427584 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427586 4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427589 4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427591 4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427594 4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427596 4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427599 4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427601 4477 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427604 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427606 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427609 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:46.429556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427611 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427614 4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427616 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427620 4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427623 4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427629 4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427632 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427634 4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427637 4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427640 4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427643 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427646 4477 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427648 4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427650 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427653 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427655 4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427658 4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427661 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427663 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:46.430170 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427665 4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427668 4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427670 4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427674 4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427676 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427678 4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427681 4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427685 4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427690 4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427693 4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427696 4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427699 4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427702 4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427704 4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427708 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427710 4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427715 4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427717 4477 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427720 4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427728 4477 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:46.430684 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427731 4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427733 4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427737 4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427742 4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427745 4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427748 4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427751 4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427753 4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427756 4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427758 4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427761 4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427763 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427766 4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427768 4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427770 4477 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427773 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427775 4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427778 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427780 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:46.431228 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427783 4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427785 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427788 4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427790 4477 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427792 4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427795 4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427797 4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427800 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427802 4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427808 4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427810 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427817 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427819 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.427827 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428390 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428396 4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428399 4477 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428402 4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428405 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428408 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:46.431705 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428410 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428413 4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428415 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428418 4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428420 4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428423 4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428425 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428427 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428430 4477 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428433 4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428436 4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428438 4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428441 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428443 4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428446 4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428448 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428451 4477 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428453 4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428456 4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428458 4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:46.432217 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428460 4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428465 4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428467 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428470 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428472 4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428476 4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428478 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428481 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428487 4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428489 4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428492 4477 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428494 4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428497 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428501 4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428505 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428507 4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428510 4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428513 4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428515 4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:46.432774 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428518 4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428520 4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428523 4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428525 4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428528 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428530 4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428533 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428535 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428538 4477 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428540 4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428543 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428546 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428548 4477 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428551 4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428555 4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428557 4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428560 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428563 4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428566 4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428569 4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:46.433258 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428572 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428575 4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428577 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428580 4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428582 4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428585 4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428587 4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428590 4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428592 4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428594 4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428597 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428599 4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428602 4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428604 4477 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428607 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428609 4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428611 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428614 4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428616 4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428619 4477 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:46.433741 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428621 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428624 4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428626 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428629 4477 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428631 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428634 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428638 4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428640 4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428643 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428648 4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428651 4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428654 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428657 4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428660 4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428663 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428665 4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428670 4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428673 4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428675 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:46.434230 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428678 4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.428680 4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428781 4477 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428792 4477 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428802 4477 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428809 4477 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428814 4477 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428817 4477 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428821 4477 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428825 4477 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428828 4477 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428831 4477 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428834 4477 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428838 4477 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428841 4477 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428844 4477 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428847 4477 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428850 4477 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428853 4477 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428858 4477 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428862 4477 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428864 4477 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428867 4477 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428870 4477 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:30:46.434725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428874 4477 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428877 4477 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428881 4477 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428884 4477 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428887 4477 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428892 4477 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428895 4477 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428897 4477 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428900 4477 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428903 4477 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428908 4477 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428911 4477 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428913 4477 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428916 4477 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428919 4477 flags.go:64] FLAG: --enable-server="true" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428922 4477 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428926 4477 flags.go:64] FLAG: --event-burst="100" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428929 4477 flags.go:64] FLAG: --event-qps="50" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428932 4477 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428935 4477 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428938 4477 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428942 4477 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428945 4477 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428948 4477 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428951 4477 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:30:46.435656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428954 4477 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428957 4477 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428961 4477 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428964 4477 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428967 4477 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428969 4477 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428972 4477 flags.go:64] FLAG: --feature-gates="" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428976 4477 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428979 4477 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428982 4477 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428986 4477 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428989 4477 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428993 4477 flags.go:64] FLAG: --help="false" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428996 4477 flags.go:64] FLAG: --hostname-override="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.428999 4477 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429002 4477 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429005 4477 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429008 4477 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429011 4477 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429014 4477 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429017 4477 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429019 4477 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429022 4477 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:30:46.436452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429025 4477 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429028 4477 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429031 4477 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429034 4477 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429036 4477 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429039 4477 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429042 4477 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429045 4477 flags.go:64] FLAG: --lock-file="" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429048 4477 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429051 4477 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429054 4477 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429060 4477 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429063 4477 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429066 4477 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429068 4477 flags.go:64] FLAG: --logging-format="text" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429087 4477 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429090 4477 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429093 4477 flags.go:64] FLAG: --manifest-url="" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429095 4477 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429100 4477 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429103 4477 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429108 4477 flags.go:64] FLAG: --max-pods="110" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429111 4477 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429113 4477 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429117 4477 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:30:46.437164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429119 4477 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429122 4477 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429125 4477 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429127 4477 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429135 4477 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429138 4477 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429140 4477 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429143 4477 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429146 4477 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429149 4477 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429152 4477 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429155 4477 flags.go:64] FLAG: --port="10250" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429158 4477 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429161 4477 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-017a2b02feea7ba52" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429164 4477 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429167 4477 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429170 4477 flags.go:64] FLAG: --register-node="true" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429173 4477 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429177 4477 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429179 4477 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429182 4477 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429185 4477 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429189 4477 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429192 4477 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429194 4477 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429197 4477 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:30:46.437792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429200 4477 flags.go:64] FLAG: --runonce="false" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429203 4477 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429208 4477 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429211 4477 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429214 4477 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429216 4477 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429219 4477 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429222 4477 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429225 4477 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429227 4477 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429230 4477 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429233 4477 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429235 4477 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429238 4477 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429241 4477 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429244 4477 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429249 4477 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429251 4477 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429254 4477 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429258 4477 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429261 4477 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429264 4477 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429266 4477 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429269 4477 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429272 4477 flags.go:64] FLAG: --v="2" Nov 01 20:30:46.438485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429276 4477 flags.go:64] FLAG: --version="false" Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429280 4477 flags.go:64] FLAG: --vmodule="" Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429284 4477 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429287 4477 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429438 4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429441 4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429444 4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429447 4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429449 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429454 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429457 4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429460 4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429463 4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429465 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429468 4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429470 4477 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429473 4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429475 4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429478 4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429480 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:46.439134 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429483 4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429486 4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429488 4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429490 4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429493 4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429495 4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429497 4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429500 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429502 4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429506 4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429509 4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429513 4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429516 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429519 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429521 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429523 4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429526 4477 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429528 4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429531 4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:46.439642 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429533 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429536 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429541 4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429544 4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429547 4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429549 4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429552 4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429554 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429557 4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429559 4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429562 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429565 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429568 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429570 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429573 4477 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429575 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429578 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429580 4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429583 4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:46.440240 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429585 4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429588 4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429590 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429593 4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429595 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429598 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429600 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429603 4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429605 4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429608 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429610 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429612 4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429615 4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429617 4477 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429620 4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429627 4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429630 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429633 4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429636 4477 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:46.440781 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429639 4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429641 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429644 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429646 4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429649 4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429651 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429654 4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429656 4477 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429659 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429661 4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429664 4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429666 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429669 4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429671 4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429674 4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429676 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429678 4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429681 4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429683 4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429686 4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:46.441290 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429688 4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429691 4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429693 4477 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429696 4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429698 4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429701 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429703 4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429706 4477 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429711 4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429713 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429716 4477 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429719 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429722 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.429750 4477 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429783 4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429787 4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:46.441780 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429790 4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429793 4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429795 4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429798 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429800 4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429805 4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429809 4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429813 4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429816 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429818 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429821 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429823 4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429826 4477 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429828 4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429831 4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429833 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429836 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429838 4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429841 4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:46.442294 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429843 4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429846 4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429848 4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429851 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429853 4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429857 4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429860 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429863 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429866 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429868 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429871 4477 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429874 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429876 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429879 4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429881 4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429883 4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429886 4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429888 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429891 4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:46.442826 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429893 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429896 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429898 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429900 4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429903 4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429905 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429908 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429910 4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429912 4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429915 4477 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429917 4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429921 4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429924 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429927 4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429930 4477 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429933 4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429936 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429939 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429944 4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:46.443325 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429947 4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429950 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429953 4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429955 4477 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429958 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429961 4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429963 4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429966 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429969 4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429972 4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429974 4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429977 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429979 4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429981 4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429984 4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429986 4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429989 4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429991 4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429993 4477 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429996 4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:46.443790 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.429998 4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430001 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430004 4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430006 4477 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430009 4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430011 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430014 4477 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430016 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430019 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430021 4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430023 4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430028 4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430032 4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430034 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430037 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430039 4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430041 4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430044 4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430051 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430054 4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:46.444420 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430057 4477 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430059 4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430062 4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430065 4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430068 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430084 4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.430086 4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.433817 4477 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.433831 4477 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.433849 4477 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.433853 4477 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433908 4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433912 4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433916 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433918 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433921 4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433924 4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433926 4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433929 4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:46.444916 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433932 4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433934 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433937 4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433939 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433943 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433946 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433948 4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433951 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433954 4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433956 4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433959 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433962 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433964 4477 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433967 4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433969 4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433972 4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433974 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433976 4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433979 4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:46.445406 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433981 4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433984 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433986 4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433988 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433991 4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433993 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433996 4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.433998 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434001 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434003 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434006 4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434008 4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434010 4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434013 4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434015 4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434018 4477 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434020 4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434022 4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434025 4477 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434027 4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:46.445937 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434030 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434032 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434036 4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434041 4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434044 4477 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434047 4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434050 4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434052 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434056 4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434059 4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434062 4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434064 4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434067 4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434082 4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434085 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434087 4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434090 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434092 4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434095 4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:46.446461 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434097 4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434100 4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434102 4477 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434105 4477 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434107 4477 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434109 4477 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434112 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434114 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434117 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434119 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434122 4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434125 4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434128 4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434130 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434132 4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434135 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434137 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434140 4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434142 4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:46.446932 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434145 4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434148 4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434150 4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434152 4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434155 4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434157 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434160 4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434162 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434164 4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434167 4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434169 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434171 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434174 4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434176 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434193 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434197 4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434200 4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434202 4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434205 4477 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434208 4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:46.447407 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434211 4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.434240 4477 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434373 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434377 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434381 4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434383 4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434386 4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434389 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434391 4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434393 4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434396 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434398 4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434401 4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434404 4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434406 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434409 4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:46.447908 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434411 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434414 4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434416 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434419 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434421 4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434423 4477 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434426 4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434428 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434431 4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434433 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434436 4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434438 4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434441 4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434452 4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434455 4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434457 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434460 4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434462 4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434465 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:46.448322 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434469 4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434472 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434477 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434479 4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434483 4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434487 4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434490 4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434492 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434495 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434497 4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434500 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434503 4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434506 4477 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434508 4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434511 4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434513 4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434516 4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434518 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434521 4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434523 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:46.448776 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434526 4477 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434528 4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434531 4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434533 4477 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434536 4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434538 4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434540 4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434543 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434545 4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434548 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434552 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434555 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434557 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434559 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434563 4477 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434566 4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434568 4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434571 4477 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434573 4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434576 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:46.449296 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434578 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434581 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434583 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434585 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434588 4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434591 4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434593 4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434596 4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434599 4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434601 4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434604 4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434606 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434609 4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434611 4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434614 4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434616 4477 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434618 4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434621 4477 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434623 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:46.449785 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434626 4477 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434628 4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434630 4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434634 4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434636 4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434639 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434641 4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434645 4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434648 4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434650 4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434653 4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434655 4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434658 4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434660 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.434683 4477 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:46.450316 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434713 4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434717 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434719 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434723 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434725 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434728 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434730 4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434733 4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434736 4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434739 4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434742 4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434744 4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434747 4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434749 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434751 4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434754 4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434757 4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434759 4477 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434761 4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:46.450730 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434764 4477 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434768 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434770 4477 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434773 4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434775 4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434778 4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434783 4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434785 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434788 4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434790 4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434793 4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434796 4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434798 4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434801 4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434803 4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434806 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434808 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434839 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434876 4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:46.451299 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434879 4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434882 4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434885 4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434982 4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434991 4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.434996 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435001 4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435007 4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435012 4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435017 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435023 4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435028 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435032 4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435037 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435041 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435051 4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435056 4477 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435060 4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435064 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435068 4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:46.451968 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435093 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435098 4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435102 4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435106 4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435111 4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435115 4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435119 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435132 4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435137 4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435141 4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435151 4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435158 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435163 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435167 4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435174 4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435179 4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435186 4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435191 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435195 4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:46.452829 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435205 4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435209 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435213 4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435217 4477 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435222 4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435226 4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435230 4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435235 4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435239 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435243 4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435248 4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435252 4477 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435256 4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435265 4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435269 4477 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435273 4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435279 4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435284 4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435288 4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435293 4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:46.453536 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435299 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435303 4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435307 4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435311 4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435325 4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435329 4477 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435333 4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435338 4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:46.435342 4477 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.435478 4477 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.436904 4477 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.436958 4477 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.437169 4477 server.go:1014] "Starting client certificate rotation" Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.437309 4477 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.437364 4477 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 20:25:05 +0000 UTC" deadline="2028-05-10 23:10:13.685353605 +0000 UTC" Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.437400 4477 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13346h39m27.247957947s" Nov 01 20:30:46.454015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.437619 4477 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:30:46.454466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.438705 4477 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:30:46.454466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.440058 4477 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:30:46.454466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.441616 4477 log.go:25] "Validated CRI v1 image API" Nov 01 20:30:46.454466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.442391 4477 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:30:46.454466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.443898 4477 fs.go:135] Filesystem UUIDs: map[34fea475-01f9-4aee-ad32-a87247e2cf85:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8451124f-b9d2-4b9f-988f-69cf9d22cf89:/dev/nvme0n1p3] Nov 01 20:30:46.454609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.443918 4477 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d2cd6301238788543f4b75f09925b0bf8fb65a15d55322b2effde78fc6ff453/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d2cd6301238788543f4b75f09925b0bf8fb65a15d55322b2effde78fc6ff453/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a90063b8a53234caa6c5762c060bc2452ed5b611c47e5e4247f570ed98d4e90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a90063b8a53234caa6c5762c060bc2452ed5b611c47e5e4247f570ed98d4e90/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e6134c20cfa5a513930172e66ad149d003de882e6fc3b296f3c66921455c6df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e6134c20cfa5a513930172e66ad149d003de882e6fc3b296f3c66921455c6df/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30d306f736cc28a60a85486a7c9aae96c3190dc0a66e4901aa6a40a35ec2520f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30d306f736cc28a60a85486a7c9aae96c3190dc0a66e4901aa6a40a35ec2520f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41e122b4e06cd1eb7e68cffdab9f77bf7fd96ed9ad187f980dbd478b60734dde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41e122b4e06cd1eb7e68cffdab9f77bf7fd96ed9ad187f980dbd478b60734dde/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5da22848387f8defe679301e3673a5e493b731195038b4002d92af918b2eb6f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5da22848387f8defe679301e3673a5e493b731195038b4002d92af918b2eb6f0/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74d487b42f799296b93df915abf0f87624765ee4b7b204d57fb30d050f7fcc62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74d487b42f799296b93df915abf0f87624765ee4b7b204d57fb30d050f7fcc62/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/810b2f22f38bcd830eed1afa324fc0e18b6e9f12cc8ee5a95e2d4e64c1215f54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/810b2f22f38bcd830eed1afa324fc0e18b6e9f12cc8ee5a95e2d4e64c1215f54/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/876b58cf27baf26f9cac87ab83427cb7cec697d9ba95b5dd1c2abbca0d8fd775/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/876b58cf27baf26f9cac87ab83427cb7cec697d9ba95b5dd1c2abbca0d8fd775/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dc3a90a021157de8a3abf5474b5abecaf830511ba7bfbbb14c9bcd118150aa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dc3a90a021157de8a3abf5474b5abecaf830511ba7bfbbb14c9bcd118150aa4/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf055069953384ea52cab63f58530cb986ab941b7040455d75eed04e2d890608/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf055069953384ea52cab63f58530cb986ab941b7040455d75eed04e2d890608/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0a5163eb731d22d1216988b4986c332b4d5c3847c664f32a387e88c72b3b0e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0a5163eb731d22d1216988b4986c332b4d5c3847c664f32a387e88c72b3b0e8/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfdf69bf90e4c3b326ae94c316fb0097303a39e91cb5cfc5a4ed3507299fcd64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfdf69bf90e4c3b326ae94c316fb0097303a39e91cb5cfc5a4ed3507299fcd64/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed4cb2e867b390c75069bf863ac23d8617aff6381efab7bb728c2e866890a20f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed4cb2e867b390c75069bf863ac23d8617aff6381efab7bb728c2e866890a20f/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5b6bcb9c3083dd890446d460731ac6860c3ef5c51f106e1bf40c0c03cf728c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5b6bcb9c3083dd890446d460731ac6860c3ef5c51f106e1bf40c0c03cf728c6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04878621-4728-4a68-8e75-1af069bec931/volumes/kubernetes.io~projected/kube-api-access-m9h84:{mountpoint:/var/lib/kubelet/pods/04878621-4728-4a68-8e75-1af069bec931/volumes/kubernetes.io~projected/kube-api-access-m9h84 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04878621-4728-4a68-8e75-1af069bec931/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/04878621-4728-4a68-8e75-1af069bec931/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09/volumes/kubernetes.io~projected/kube-api-access-dznxf:{mountpoint:/var/lib/kubelet/pods/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09/volumes/kubernetes.io~projected/kube-api-access-dznxf major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1544a0e3-068d-4d70-bbfb-df3e4f86f391/volumes/kubernetes.io~projected/kube-api-access-bhrpr:{mountpoint:/var/lib/kubelet/pods/1544a0e3-068d-4d70-bbfb-df3e4f86f391/volumes/kubernetes.io~projected/kube-api-access-bhrpr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e69a8f9-67ff-4ee6-9718-ab2c186021a1/volumes/kubernetes.io~projected/kube-api-access-bmx8q:{mountpoint:/var/lib/kubelet/pods/2e69a8f9-67ff-4ee6-9718-ab2c186021a1/volumes/kubernetes.io~projected/kube-api-access-bmx8q major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a69377-a3ea-42ab-8a85-753770827b97/volumes/kubernetes.io~projected/kube-api-access-tcjb6:{mountpoint:/var/lib/kubelet/pods/40a69377-a3ea-42ab-8a85-753770827b97/volumes/kubernetes.io~projected/kube-api-access-tcjb6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4220aacf-5c03-4983-904f-6b0d522cb8c6/volumes/kubernetes.io~projected/kube-api-access-64fm9:{mountpoint:/var/lib/kubelet/pods/4220aacf-5c03-4983-904f-6b0d522cb8c6/volumes/kubernetes.io~projected/kube-api-access-64fm9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65662b67-e317-41c9-959e-24c40741d793/volumes/kubernetes.io~projected/kube-api-access-852xh:{mountpoint:/var/lib/kubelet/pods/65662b67-e317-41c9-959e-24c40741d793/volumes/kubernetes.io~projected/kube-api-access-852xh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/689471e9-1161-4ae7-b08a-15b85ffaa359/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/689471e9-1161-4ae7-b08a-15b85ffaa359/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d9a1fb0-d057-4922-97a5-62ea9bebf833/volumes/kubernetes.io~projected/kube-api-access-j5mqs:{mountpoint:/var/lib/kubelet/pods/6d9a1fb0-d057-4922-97a5-62ea9bebf833/volumes/kubernetes.io~projected/kube-api-access-j5mqs major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96533abd-b0c9-46eb-bf51-96e4c58da84d/volumes/kubernetes.io~projected/kube-api-access-fqbjz:{mountpoint:/var/lib/kubelet/pods/96533abd-b0c9-46eb-bf51-96e4c58da84d/volumes/kubernetes.io~projected/kube-api-access-fqbjz major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~projected/kube-api-access-s9z69:{mountpoint:/var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~projected/kube-api-access-s9z69 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ace2d1f7-e48a-4375-848d-4cd0f9122046/volumes/kubernetes.io~projected/kube-api-access-nlt4b:{mountpoint:/var/lib/kubelet/pods/ace2d1f7-e48a-4375-848d-4cd0f9122046/volumes/kubernetes.io~projected/kube-api-access-nlt4b major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volumes/kubernetes.io~projected/kube-api-access-h6v8m:{mountpoint:/var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volumes/kubernetes.io~projected/kube-api-access-h6v8m major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c481a350-2dc5-450b-b59c-2a37640d458e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c481a350-2dc5-450b-b59c-2a37640d458e/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9d566de-6bf8-4fc4-966b-ac984e4a300e/volumes/kubernetes.io~projected/kube-api-access-b27cw:{mountpoint:/var/lib/kubelet/pods/e9d566de-6bf8-4fc4-966b-ac984e4a300e/volumes/kubernetes.io~projected/kube-api-access-b27cw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f62b13d3-9a85-404d-b7c5-0a482d942bf3/volumes/kubernetes.io~projected/kube-api-access-9tbhw:{mountpoint:/var/lib/kubelet/pods/f62b13d3-9a85-404d-b7c5-0a482d942bf3/volumes/kubernetes.io~projected/kube-api-access-9tbhw major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6d29d9e-1400-40e2-b20b-793790117269/volumes/kubernetes.io~projected/kube-api-access-tqxvs:{mountpoint:/var/lib/kubelet/pods/f6d29d9e-1400-40e2-b20b-793790117269/volumes/kubernetes.io~projected/kube-api-access-tqxvs major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/97f953f26b974dad8d02d368d48c28b75dbf53338c7b4351a73d3d24344a988b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/32f98cf5c31a577bc6cbe75f31a470abe7c3f46632c3a87028bdea65d0630564/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/853369dca8264a54b90cff431ba89b788d65a26e137c7f18e175655dc3db032e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b636a2d6704b8ecaaf9e561b8633ff7a03769f1f81fca49e72055ee2fff31960/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f2f144d73fa86f7c7bdd6c333a994555f3a08b8a0bb67a510a91017a69af3d66/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/03d3e1f41109b99ecc4b7a911dfff73dd85b00e898dca0787daab3fe6162bd8a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9bea1949767515d9ec11b87b083d0b670eff98b4d3ee5658c17b0dbf3d58028a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/451c7f24fb8a3a9a0c4882641babdbbfba32c40e2dc7ba4c247d195c1b945361/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4e401eb12376447b29724ecdeaf18edcd2d2f2b750124ef9dfaff50140d40bca/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/62b449dee7b57e6932c1964d2fd0696378d47901a4e93688e7ff0645d63b5370/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/5a73449d1c85d7d555d5c812525bf1e33a484213cfe0dc57e033ff2b9314b6fb/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/2d689829d896d8ee6d87cbb227faffdc8240646ae8b94b40a644d7e19f742a7e/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/cba572168742efb12ae7cb526fee231c625dcfc2939ef881885fb12dd59dce0a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/eb4e2b7e7ce113ac403f52b02bb17f141d95463d8fcc2958f53258ce995f3f4c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/df63caa0705aa6c1cce4f830fb9a3b04a1e37b1e14275efe20a58033849d2eff/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b628531412395dfcc372cb3d3b26c86b3346b46afe188f312ea77d3cadce1172/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/06ada6798f34eb1f876f3abbb4acb350d2d24e8ea76cf73e3489dbca9b22ac1b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/57e6b1eb33389e24ae634a6cdd1cd4d927ed27f10cf9782ce2661768d5e1580f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ed5549a40210cc9870704aed6a75b1dd123cbf4153b9e3c33a2e1fe64858d506/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/78cfefd1088124b674768553cf76079997937fda2ff9f3d2be2015dd250ee2c8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/80fd64b13314f88e40bbb8c4df53cc75af7445330a4ac8bb3ed427c9d4de50e0/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/dfc576897a363f84dbc6b859ae4e849b0281f5aafe8f16b094c25f073f379e2c/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/122e0161db4c3e968cbad5354c11f0ebf90fbb6f67d2cfc369ead9c2e878b3e2/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/1bc1adbd4acdedd1e68638a1632734ba131926f8920f4a005dfcd43271f7753b/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/143dc5279bb201d87d46f9e127ccfbc3cda69ac337aaf69f6df544d1115dc288/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/6d0ee2678fb7844931e3ebb7cdb1c97e9c6847f8882b321137b0593eb9e2b4f6/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/126a1491fec5448e100b710eb623543c9716ccdbcc0c2e34570e3f2f153b84f0/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/b4c370dc1681bfb0bde1b1c5abefb6bd83635624fe57c2bd5d714dfd57a3bcb8/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/1d26abed8dd45b286c218a88a49db80d729132d965d8ebb92b1469a92f99a9c9/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/1d1c75241257d4f86183945d379580159c8a4b390af0b92c399d8e19dfda01eb/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/3f1198c579f18023dac036c5b139a594f59c34c79f41f89626f655c2702c4b33/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2fe8526103858000fcde76b96e59cfc52617a2fb3c17f5a4144c9cad536b648f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/998c8b8fbaed193a05f015a11cc63af695c1b71e686954940d8ee739d527f1de/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a5fe6b86ae678da13d3f9f060542d370751f6da8365f1d6f326e0d0b9df58cc9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ea0f3fe960c497d9f7bbef89ee1a0132b0f7aa21411acc2205d8e4a1a315814d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/432bc1d0c8bb0065ca1dcec3a5d2339e6d0c4eb71455e0f50bbf3e17aa5977f1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/5a1e0d656d425468a7e5575efc062fafb12f14ed2f703363abbe83f8e811d5d0/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/cf33ae4ff1a302ca41597a94d9bb35f8be7a47a9e8bdf8bc3a57aef52c7f90dd/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/104d99d946cbbd3ea5de1a2f897dce5c2f0d5ccd639345eb45f48f410cf49a50/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4b6343790b265c1f0d54d78bbef1fcdccc9017432be1f8e4633580635242f38d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c3cd69e13bf77bb61a2308132348039baff4d0c7203e48ca8509a04758f34076/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.449676 4477 manager.go:217] Machine: {Timestamp:2026-11-01 20:30:46.449227433 +0000 UTC m=+0.099278968 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099616 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24cd7457da3e6b238f00f217bf14a0 SystemUUID:ec24cd74-57da-3e6b-238f-00f217bf14a0 BootID:4dcfa89e-c429-44ac-acc4-42c1bae4a3a9 Filesystems:[{Device:/var/lib/kubelet/pods/65662b67-e317-41c9-959e-24c40741d793/volumes/kubernetes.io~projected/kube-api-access-852xh DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~projected/kube-api-access-s9z69 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ace2d1f7-e48a-4375-848d-4cd0f9122046/volumes/kubernetes.io~projected/kube-api-access-nlt4b DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1544a0e3-068d-4d70-bbfb-df3e4f86f391/volumes/kubernetes.io~projected/kube-api-access-bhrpr DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e69a8f9-67ff-4ee6-9718-ab2c186021a1/volumes/kubernetes.io~projected/kube-api-access-bmx8q DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d2cd6301238788543f4b75f09925b0bf8fb65a15d55322b2effde78fc6ff453/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30d306f736cc28a60a85486a7c9aae96c3190dc0a66e4901aa6a40a35ec2520f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f62b13d3-9a85-404d-b7c5-0a482d942bf3/volumes/kubernetes.io~projected/kube-api-access-9tbhw DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volumes/kubernetes.io~projected/kube-api-access-h6v8m DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfdf69bf90e4c3b326ae94c316fb0097303a39e91cb5cfc5a4ed3507299fcd64/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6d29d9e-1400-40e2-b20b-793790117269/volumes/kubernetes.io~projected/kube-api-access-tqxvs DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5da22848387f8defe679301e3673a5e493b731195038b4002d92af918b2eb6f0/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c481a350-2dc5-450b-b59c-2a37640d458e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5b6bcb9c3083dd890446d460731ac6860c3ef5c51f106e1bf40c0c03cf728c6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41e122b4e06cd1eb7e68cffdab9f77bf7fd96ed9ad187f980dbd478b60734dde/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4220aacf-5c03-4983-904f-6b0d522cb8c6/volumes/kubernetes.io~projected/kube-api-access-64fm9 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74d487b42f799296b93df915abf0f87624765ee4b7b204d57fb30d050f7fcc62/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/876b58cf27baf26f9cac87ab83427cb7cec697d9ba95b5dd1c2abbca0d8fd775/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/96533abd-b0c9-46eb-bf51-96e4c58da84d/volumes/kubernetes.io~projected/kube-api-access-fqbjz DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09/volumes/kubernetes.io~projected/kube-api-access-dznxf DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9d566de-6bf8-4fc4-966b-ac984e4a300e/volumes/kubernetes.io~projected/kube-api-access-b27cw DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/04878621-4728-4a68-8e75-1af069bec931/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dc3a90a021157de8a3abf5474b5abecaf830511ba7bfbbb14c9bcd118150aa4/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf055069953384ea52cab63f58530cb986ab941b7040455d75eed04e2d890608/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/810b2f22f38bcd830eed1afa324fc0e18b6e9f12cc8ee5a95e2d4e64c1215f54/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d633766-ee45-4299-a89f-7db3cd012d43/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04878621-4728-4a68-8e75-1af069bec931/volumes/kubernetes.io~projected/kube-api-access-m9h84 DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6d9a1fb0-d057-4922-97a5-62ea9bebf833/volumes/kubernetes.io~projected/kube-api-access-j5mqs DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/689471e9-1161-4ae7-b08a-15b85ffaa359/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a90063b8a53234caa6c5762c060bc2452ed5b611c47e5e4247f570ed98d4e90/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40a69377-a3ea-42ab-8a85-753770827b97/volumes/kubernetes.io~projected/kube-api-access-tcjb6 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0a5163eb731d22d1216988b4986c332b4d5c3847c664f32a387e88c72b3b0e8/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bade2e15-5673-4f0e-9fa4-fec7d544507c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e6134c20cfa5a513930172e66ad149d003de882e6fc3b296f3c66921455c6df/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed4cb2e867b390c75069bf863ac23d8617aff6381efab7bb728c2e866890a20f/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1a90063b8a53234 MacAddress:aa:4d:cc:c0:4b:bc Speed:10000 Mtu:8901} {Name:876b58cf27baf26 MacAddress:56:b1:71:b0:3a:75 Speed:10000 Mtu:8901} {Name:9dc3a90a021157d MacAddress:be:69:78:68:74:e1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:38:1d:8b:88:75 Speed:0 Mtu:9001} {Name:br-int MacAddress:9e:7f:0a:55:83:ce Speed:0 Mtu:8901} {Name:cf055069953384e MacAddress:12:d6:84:15:6f:3a Speed:10000 Mtu:8901} {Name:ed4cb2e867b390c MacAddress:ea:37:49:d5:d8:27 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:38:1d:8b:88:75 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:02:34:81:e2:31:96 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:b7:47:a9:3e:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.449909 4477 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.449947 4477 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450210 4477 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450232 4477 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-242.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450347 4477 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450354 4477 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450365 4477 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450379 4477 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450479 4477 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450634 4477 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450750 4477 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450785 4477 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450798 4477 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450808 4477 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450818 4477 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.450826 4477 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.452364 4477 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.452379 4477 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:30:46.454762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.453217 4477 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:30:46.455848 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.455832 4477 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:30:46.455882 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.455851 4477 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:30:46.455962 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.455951 4477 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 20:30:46.456237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456222 4477 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:30:46.456280 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456269 4477 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:30:46.456385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456375 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:30:46.456413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456395 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:30:46.456413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456404 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:30:46.456467 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456414 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:30:46.456467 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456423 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:30:46.456467 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456432 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:30:46.456467 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456442 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:30:46.456467 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456452 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:30:46.456467 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456462 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:30:46.456656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456470 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:30:46.456656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456483 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:30:46.456656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456496 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:30:46.456656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456532 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:30:46.456656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.456540 4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:30:46.459825 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.459812 4477 server.go:1263] "Started kubelet" Nov 01 20:30:46.459919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.459890 4477 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:30:46.459984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.459905 4477 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:30:46.460379 ip-10-0-133-242 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:30:46.460674 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.460523 4477 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:30:46.460895 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.460878 4477 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:30:46.461099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.461066 4477 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:30:46.462856 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.462837 4477 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.463212 4477 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.463275 4477 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:07 +0000 UTC" deadline="2028-05-09 17:43:13.864422941 +0000 UTC" Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.463292 4477 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13317h12m27.401133706s" Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.463440 4477 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.463451 4477 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.463555 4477 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.463573 4477 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.464090 4477 factory.go:55] Registering systemd factory Nov 01 20:30:46.464769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.464107 4477 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465591 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65662b67-e317-41c9-959e-24c40741d793" volumeName="kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465625 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d9a1fb0-d057-4922-97a5-62ea9bebf833" volumeName="kubernetes.io/empty-dir/6d9a1fb0-d057-4922-97a5-62ea9bebf833-tmp-dir" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465640 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6d29d9e-1400-40e2-b20b-793790117269" volumeName="kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465653 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6d29d9e-1400-40e2-b20b-793790117269" volumeName="kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465665 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bade2e15-5673-4f0e-9fa4-fec7d544507c" volumeName="kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-env-overrides" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465678 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e69a8f9-67ff-4ee6-9718-ab2c186021a1" volumeName="kubernetes.io/empty-dir/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-tmp-dir" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465690 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4220aacf-5c03-4983-904f-6b0d522cb8c6" volumeName="kubernetes.io/projected/4220aacf-5c03-4983-904f-6b0d522cb8c6-kube-api-access-64fm9" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465701 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d9a1fb0-d057-4922-97a5-62ea9bebf833" volumeName="kubernetes.io/projected/6d9a1fb0-d057-4922-97a5-62ea9bebf833-kube-api-access-j5mqs" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465715 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96533abd-b0c9-46eb-bf51-96e4c58da84d" volumeName="kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465726 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ace2d1f7-e48a-4375-848d-4cd0f9122046" volumeName="kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465740 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ace2d1f7-e48a-4375-848d-4cd0f9122046" volumeName="kubernetes.io/empty-dir/ace2d1f7-e48a-4375-848d-4cd0f9122046-data-volume" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465753 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a69377-a3ea-42ab-8a85-753770827b97" volumeName="kubernetes.io/configmap/40a69377-a3ea-42ab-8a85-753770827b97-serviceca" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465764 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65662b67-e317-41c9-959e-24c40741d793" volumeName="kubernetes.io/projected/65662b67-e317-41c9-959e-24c40741d793-kube-api-access-852xh" seLinuxMountContext="" Nov 01 20:30:46.465669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465776 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d633766-ee45-4299-a89f-7db3cd012d43" volumeName="kubernetes.io/projected/9d633766-ee45-4299-a89f-7db3cd012d43-kube-api-access-s9z69" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465791 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d566de-6bf8-4fc4-966b-ac984e4a300e" volumeName="kubernetes.io/projected/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kube-api-access-b27cw" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465803 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04878621-4728-4a68-8e75-1af069bec931" volumeName="kubernetes.io/projected/04878621-4728-4a68-8e75-1af069bec931-kube-api-access-m9h84" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465816 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d9a1fb0-d057-4922-97a5-62ea9bebf833" volumeName="kubernetes.io/configmap/6d9a1fb0-d057-4922-97a5-62ea9bebf833-config-volume" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465827 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bade2e15-5673-4f0e-9fa4-fec7d544507c" volumeName="kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-script-lib" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465839 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c481a350-2dc5-450b-b59c-2a37640d458e" volumeName="kubernetes.io/secret/c481a350-2dc5-450b-b59c-2a37640d458e-agent-certs" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465851 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c893f01-c22e-4e7d-a4ce-8e7896ef0e09" volumeName="kubernetes.io/projected/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09-kube-api-access-dznxf" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465862 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4220aacf-5c03-4983-904f-6b0d522cb8c6" volumeName="kubernetes.io/configmap/4220aacf-5c03-4983-904f-6b0d522cb8c6-iptables-alerter-script" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465873 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65662b67-e317-41c9-959e-24c40741d793" volumeName="kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-binary-copy" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465895 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d633766-ee45-4299-a89f-7db3cd012d43" volumeName="kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-etc-tuned" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465906 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ace2d1f7-e48a-4375-848d-4cd0f9122046" volumeName="kubernetes.io/projected/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-api-access-nlt4b" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465917 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c481a350-2dc5-450b-b59c-2a37640d458e" volumeName="kubernetes.io/configmap/c481a350-2dc5-450b-b59c-2a37640d458e-konnectivity-ca" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465929 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f62b13d3-9a85-404d-b7c5-0a482d942bf3" volumeName="kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-daemon-config" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465942 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f62b13d3-9a85-404d-b7c5-0a482d942bf3" volumeName="kubernetes.io/projected/f62b13d3-9a85-404d-b7c5-0a482d942bf3-kube-api-access-9tbhw" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465954 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1544a0e3-068d-4d70-bbfb-df3e4f86f391" volumeName="kubernetes.io/projected/1544a0e3-068d-4d70-bbfb-df3e4f86f391-kube-api-access-bhrpr" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465971 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a69377-a3ea-42ab-8a85-753770827b97" volumeName="kubernetes.io/projected/40a69377-a3ea-42ab-8a85-753770827b97-kube-api-access-tcjb6" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.465982 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bade2e15-5673-4f0e-9fa4-fec7d544507c" volumeName="kubernetes.io/secret/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466006 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f62b13d3-9a85-404d-b7c5-0a482d942bf3" volumeName="kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cni-binary-copy" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466017 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6d29d9e-1400-40e2-b20b-793790117269" volumeName="kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466029 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65662b67-e317-41c9-959e-24c40741d793" volumeName="kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466040 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="689471e9-1161-4ae7-b08a-15b85ffaa359" volumeName="kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466053 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d633766-ee45-4299-a89f-7db3cd012d43" volumeName="kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-tmp" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466064 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bade2e15-5673-4f0e-9fa4-fec7d544507c" volumeName="kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-config" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466098 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bade2e15-5673-4f0e-9fa4-fec7d544507c" volumeName="kubernetes.io/projected/bade2e15-5673-4f0e-9fa4-fec7d544507c-kube-api-access-h6v8m" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466110 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6d29d9e-1400-40e2-b20b-793790117269" volumeName="kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466123 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04878621-4728-4a68-8e75-1af069bec931" volumeName="kubernetes.io/secret/04878621-4728-4a68-8e75-1af069bec931-cert" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466133 4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e69a8f9-67ff-4ee6-9718-ab2c186021a1" volumeName="kubernetes.io/projected/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-kube-api-access-bmx8q" seLinuxMountContext="" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466146 4477 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:30:46.466501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.466157 4477 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:30:46.468241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.468195 4477 factory.go:153] Registering CRI-O factory Nov 01 20:30:46.468241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.468217 4477 factory.go:223] Registration of the crio container factory successfully Nov 01 20:30:46.469036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.468272 4477 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:30:46.469036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.468289 4477 factory.go:103] Registering Raw factory Nov 01 20:30:46.469036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.468306 4477 manager.go:1199] Started watching for new ooms in manager Nov 01 20:30:46.469036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.468630 4477 manager.go:319] Starting recovery of all containers Nov 01 20:30:46.494829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.494766 4477 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:30:46.496043 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.496022 4477 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:30:46.496043 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.496045 4477 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:30:46.496197 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.496065 4477 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:30:46.496197 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.496129 4477 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 20:30:46.505216 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.505183 4477 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="ceb8eb59e36a638da26c1588587b643f73107c67655864431c1aa7aee870b7a9" exitCode=0 Nov 01 20:30:46.505216 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.505212 4477 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="b367291c4c1f005cd4ac8a768ba2edb97d05d5cb45c23680a97723fa751d7af2" exitCode=0 Nov 01 20:30:46.505216 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.505218 4477 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="233bbfb1b884b2175d01b7ce1fe61e3e02e3cc06aa659df76129c0d561de03ea" exitCode=0 Nov 01 20:30:46.505424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.505224 4477 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="095ed284e981ac674fad2faf275e548098f850bb742af423903cafc6947865a0" exitCode=0 Nov 01 20:30:46.505424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.505230 4477 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="6f8fc19beb902db8a94c91c1facf2deb2ac00c255a3d3814fdf3d8c13169379f" exitCode=0 Nov 01 20:30:46.505424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.505234 4477 generic.go:356] "Generic (PLEG): container finished" podID="65662b67-e317-41c9-959e-24c40741d793" containerID="6a60f1ee0718e31475ec4324fed468dbf3cd14b82523869cd4adb12898560830" exitCode=0 Nov 01 20:30:46.531584 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.531551 4477 generic.go:356] "Generic (PLEG): container finished" podID="4758d5c314bffde2671732f370e32950" containerID="967e1550fed74dae51d22fcbd074ae693f8957537b662581cecf8ea88c0bad9e" exitCode=0 Nov 01 20:30:46.534673 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.534652 4477 manager.go:324] Recovery completed Nov 01 20:30:46.546756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.546735 4477 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:46.547781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.547767 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:46.547852 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.547797 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:46.547852 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.547810 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:46.549169 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549153 4477 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:30:46.549169 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549163 4477 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:46.549169 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549167 4477 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:30:46.549361 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549186 4477 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:30:46.549361 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549329 4477 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 20:30:46.549361 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549344 4477 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 20:30:46.549496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549438 4477 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 20:30:46.549496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549448 4477 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 20:30:46.549496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549456 4477 policy_none.go:50] "Start" Nov 01 20:30:46.549496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549469 4477 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:30:46.549496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549482 4477 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:30:46.549731 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549655 4477 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 20:30:46.549731 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549665 4477 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 20:30:46.549731 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549674 4477 policy_none.go:44] "Start" Nov 01 20:30:46.550008 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.549993 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:46.550065 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.550023 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:46.550065 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.550036 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:46.553782 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.553738 4477 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 20:30:46.559711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.559692 4477 manager.go:341] "Starting Device Plugin manager" Nov 01 20:30:46.559803 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.559720 4477 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 20:30:46.559803 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.559742 4477 server.go:87] "Starting device plugin registration server" Nov 01 20:30:46.560035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.560012 4477 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 20:30:46.560110 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.560027 4477 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 20:30:46.560168 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.560109 4477 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 20:30:46.560223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.560191 4477 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 20:30:46.560269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.560222 4477 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 20:30:46.560269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.560229 4477 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 20:30:46.560356 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.560285 4477 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:30:46.560213937Z","UUID":"48bbba93-c076-4819-adf5-8a872b9b05b2","Handler":null,"Name":"","Endpoint":""} Nov 01 20:30:46.561543 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.561494 4477 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 20:30:46.561647 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.561550 4477 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 20:30:46.561706 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.561679 4477 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 20:30:46.561763 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.561714 4477 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" Nov 01 20:30:46.596954 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.596534 4477 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal"] Nov 01 20:30:46.597042 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.597032 4477 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:46.597857 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.597834 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:46.597857 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.597860 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:46.597991 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.597870 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:46.598120 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.598107 4477 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:46.598197 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.598184 4477 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:46.598826 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.598810 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:46.598880 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.598838 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:46.598880 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.598849 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:46.598951 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.598810 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:46.598951 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.598911 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:46.598951 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.598922 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:46.599040 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.598966 4477 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.599202 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599172 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"fb754e3320d8dad235549d1c4b2afdd3","Type":"ContainerStarted","Data":"407a169d6b4af653460557d538b6122ed4f567ca9b71bd01558e1fc444075ea2"} Nov 01 20:30:46.599249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599218 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"fb754e3320d8dad235549d1c4b2afdd3","Type":"ContainerStarted","Data":"5da22848387f8defe679301e3673a5e493b731195038b4002d92af918b2eb6f0"} Nov 01 20:30:46.599280 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599274 4477 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:46.599317 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599287 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"4758d5c314bffde2671732f370e32950","Type":"ContainerStarted","Data":"1109f1eadff317845457dd22230493412d53de0fa518074e18a416476f8d919f"} Nov 01 20:30:46.599317 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599311 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"4758d5c314bffde2671732f370e32950","Type":"ContainerDied","Data":"967e1550fed74dae51d22fcbd074ae693f8957537b662581cecf8ea88c0bad9e"} Nov 01 20:30:46.599371 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599332 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" event={"ID":"4758d5c314bffde2671732f370e32950","Type":"ContainerStarted","Data":"810b2f22f38bcd830eed1afa324fc0e18b6e9f12cc8ee5a95e2d4e64c1215f54"} Nov 01 20:30:46.599919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599900 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:46.599965 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599921 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:46.599965 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.599930 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:46.600041 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.600031 4477 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-242.us-west-1.compute.internal\" not found" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.611627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.611603 4477 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:46.660270 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.660245 4477 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:46.661266 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.661244 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:46.661385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.661275 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:46.661385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.661290 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:46.661385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.661296 4477 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:46.661385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.661313 4477 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.661508 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.661442 4477 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 20:30:46.664600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.664586 4477 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.711280 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.711212 4477 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:46.711452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.711435 4477 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:46.758710 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.758678 4477 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:46.758846 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.758725 4477 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.758846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.758739 4477 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.761855 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.761832 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.761961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.761859 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.761961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.761881 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb754e3320d8dad235549d1c4b2afdd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"fb754e3320d8dad235549d1c4b2afdd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.810014 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.809984 4477 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.810182 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.810056 4477 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.810679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.810651 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeNotReady" Nov 01 20:30:46.810801 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.810685 4477 setters.go:546] "Node became not ready" node="ip-10-0-133-242.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T20:30:46Z","lastTransitionTime":"2026-11-01T20:30:46Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 01 20:30:46.814065 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.814049 4477 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:46.814143 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:46.814109 4477 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.821501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.821473 4477 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-242.us-west-1.compute.internal" is owned by different node (oldNodeID="e2d98a79-ac8d-45ce-b217-6c1f56c5022a", newNodeID=""), cleaning up... Nov 01 20:30:46.862069 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.862041 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.862249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.862092 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.862249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.862144 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.862249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.862150 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4758d5c314bffde2671732f370e32950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"4758d5c314bffde2671732f370e32950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.862249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.862182 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb754e3320d8dad235549d1c4b2afdd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"fb754e3320d8dad235549d1c4b2afdd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:46.862249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:46.862210 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb754e3320d8dad235549d1c4b2afdd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal\" (UID: \"fb754e3320d8dad235549d1c4b2afdd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:47.079162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.079135 4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-242.us-west-1.compute.internal" event="NodeReady" Nov 01 20:30:47.079299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.079210 4477 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 20:30:47.451846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.451776 4477 apiserver.go:52] "Watching apiserver" Nov 01 20:30:47.533937 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.533915 4477 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:47.534064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.534044 4477 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:47.554687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.554669 4477 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:30:47.555090 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.555052 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-nrcnn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k","openshift-network-operator/iptables-alerter-kdh6b","openshift-insights/insights-runtime-extractor-xtldx","openshift-cluster-node-tuning-operator/tuned-mqgcd","kube-system/konnectivity-agent-9msm2","openshift-dns/node-resolver-2lphr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal","openshift-insights/periodic-gathering-8mzmm-bpj4x","openshift-multus/network-metrics-daemon-mtclz","openshift-ovn-kubernetes/ovnkube-node-hrnnn","openshift-dns/dns-default-w9xfp","openshift-multus/multus-cq22n","openshift-multus/multus-additional-cni-plugins-zp6f6","openshift-network-diagnostics/network-check-target-sgmhx","openshift-ingress-canary/ingress-canary-bkh8x","openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn","kube-system/global-pull-secret-syncer-658ds"] Nov 01 20:30:47.561065 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.561045 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:47.561867 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.561847 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.562938 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.562910 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.565312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.565295 4477 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 20:30:47.567713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567695 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6v8m\" (UniqueName: \"kubernetes.io/projected/bade2e15-5673-4f0e-9fa4-fec7d544507c-kube-api-access-h6v8m\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.567817 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567728 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-etc-tuned\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.567817 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567757 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-hosts-file\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:47.567817 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567781 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.567817 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567807 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-kubelet\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.567961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567830 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-socket-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.567961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567853 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.567961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567829 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-etc-tuned\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.567961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567880 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-run\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.568087 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567961 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-lib-modules\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.568087 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.567998 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmx8q\" (UniqueName: \"kubernetes.io/projected/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-kube-api-access-bmx8q\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:47.568087 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568028 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-cnibin\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.568087 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568051 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-node-log\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.568198 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568104 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-bin\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.568198 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568131 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcjb6\" (UniqueName: \"kubernetes.io/projected/40a69377-a3ea-42ab-8a85-753770827b97-kube-api-access-tcjb6\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:47.568198 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568154 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.568284 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568199 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-registration-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.568284 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568265 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-sys\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.568340 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568312 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-tmp-dir\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:47.568451 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568345 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.568451 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568382 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9a1fb0-d057-4922-97a5-62ea9bebf833-tmp-dir\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.568451 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568396 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-tmp-dir\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:47.568451 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568435 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-var-lib-kubelet\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.568581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568460 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9a1fb0-d057-4922-97a5-62ea9bebf833-tmp-dir\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.568581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568469 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ace2d1f7-e48a-4375-848d-4cd0f9122046-crio-socket\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.568581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568494 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-netns\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.568581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568544 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-etc-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.568686 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568580 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-config\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.568686 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568608 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-hostroot\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.568686 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568653 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9z69\" (UniqueName: \"kubernetes.io/projected/9d633766-ee45-4299-a89f-7db3cd012d43-kube-api-access-s9z69\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.568686 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568670 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:47.568793 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568698 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-netns\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.568793 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568713 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-daemon-config\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.568793 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568727 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a69377-a3ea-42ab-8a85-753770827b97-host\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:47.568793 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568776 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.568901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568806 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlt4b\" (UniqueName: \"kubernetes.io/projected/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-api-access-nlt4b\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.568901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568830 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-systemd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.568901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568846 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cni-binary-copy\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.568901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568873 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.569006 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568927 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-systemd-units\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.569006 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568948 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-conf\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.569006 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568962 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-kubelet-config\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:47.569006 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568979 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-multus-certs\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.569006 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.568993 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-tmp\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.569166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569026 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ace2d1f7-e48a-4375-848d-4cd0f9122046-data-volume\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.569166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569061 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d633766-ee45-4299-a89f-7db3cd012d43-tmp\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.569166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569106 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-var-lib-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.569166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569130 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cnibin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.569166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569134 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ace2d1f7-e48a-4375-848d-4cd0f9122046-data-volume\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.569166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569145 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-k8s-cni-cncf-io\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.569352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569194 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-multus\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.569352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569219 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mqs\" (UniqueName: \"kubernetes.io/projected/6d9a1fb0-d057-4922-97a5-62ea9bebf833-kube-api-access-j5mqs\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.569352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569238 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64fm9\" (UniqueName: \"kubernetes.io/projected/4220aacf-5c03-4983-904f-6b0d522cb8c6-kube-api-access-64fm9\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:47.569352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569267 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:47.569352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569324 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-slash\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.569551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569370 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:47.569551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569398 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1fb0-d057-4922-97a5-62ea9bebf833-config-volume\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.569551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569425 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysconfig\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.569551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569467 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznxf\" (UniqueName: \"kubernetes.io/projected/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09-kube-api-access-dznxf\") pod \"migrator-66df5d496b-kshpn\" (UID: \"0c893f01-c22e-4e7d-a4ce-8e7896ef0e09\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" Nov 01 20:30:47.569551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569489 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.569551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569518 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.569551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569541 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04878621-4728-4a68-8e75-1af069bec931-cert\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:47.569846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569581 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:47.569846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569609 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.569846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569633 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-bin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.569846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569668 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-sys-fs\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.569846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569745 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27cw\" (UniqueName: \"kubernetes.io/projected/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kube-api-access-b27cw\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.569846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569813 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.569846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569839 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852xh\" (UniqueName: \"kubernetes.io/projected/65662b67-e317-41c9-959e-24c40741d793-kube-api-access-852xh\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569857 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569910 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569936 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.569955 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-etc-kubernetes\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570042 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4220aacf-5c03-4983-904f-6b0d522cb8c6-host-slash\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570068 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-modprobe-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570108 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9h84\" (UniqueName: \"kubernetes.io/projected/04878621-4728-4a68-8e75-1af069bec931-kube-api-access-m9h84\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570132 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-env-overrides\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.570179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570182 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-system-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570198 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570212 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-socket-dir-parent\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570229 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40a69377-a3ea-42ab-8a85-753770827b97-serviceca\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570244 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570259 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570287 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-conf-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570308 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrpr\" (UniqueName: \"kubernetes.io/projected/1544a0e3-068d-4d70-bbfb-df3e4f86f391-kube-api-access-bhrpr\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570372 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-dbus\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570402 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-ovn\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570420 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.570456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570456 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4220aacf-5c03-4983-904f-6b0d522cb8c6-iptables-alerter-script\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:47.570738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570489 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-kubernetes\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.570738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570504 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-system-cni-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.570738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570548 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxvs\" (UniqueName: \"kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:47.570738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570573 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-netd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.570738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570600 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-script-lib\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.570738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570669 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbhw\" (UniqueName: \"kubernetes.io/projected/f62b13d3-9a85-404d-b7c5-0a482d942bf3-kube-api-access-9tbhw\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.570738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570688 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-device-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.570738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570715 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-systemd\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.570998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570805 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-host\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.570998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570827 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c481a350-2dc5-450b-b59c-2a37640d458e-agent-certs\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:47.570998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570853 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c481a350-2dc5-450b-b59c-2a37640d458e-konnectivity-ca\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:47.570998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570888 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-kubelet\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.570998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570925 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-log-socket\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.570998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.570942 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovn-node-metrics-cert\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.620432 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.620412 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:30:47.620432 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.620413 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pjdf7\"" Nov 01 20:30:47.620619 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.620602 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h288r\"" Nov 01 20:30:47.620824 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.620813 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9xjd\"" Nov 01 20:30:47.620973 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.620955 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bdld\"" Nov 01 20:30:47.621219 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.621206 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c481a350-2dc5-450b-b59c-2a37640d458e-agent-certs\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:47.623641 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.623622 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.623726 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.623648 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:30:47.623726 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.623676 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.625309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.625292 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:30:47.628331 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.628317 4477 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:47.628374 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:47.628352 4477 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:47.629097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.629067 4477 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:47.629146 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:47.629114 4477 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-242.us-west-1.compute.internal" Nov 01 20:30:47.629250 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.629234 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cni-binary-copy\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.630362 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.630347 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.631337 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.631322 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c481a350-2dc5-450b-b59c-2a37640d458e-konnectivity-ca\") pod \"konnectivity-agent-9msm2\" (UID: \"c481a350-2dc5-450b-b59c-2a37640d458e\") " pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:30:47.671149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671129 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.671232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671157 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.671232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671174 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-conf-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.671232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671193 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-dbus\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:47.671232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671213 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-ovn\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671235 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671250 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-conf-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671269 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671278 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-dbus\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671238 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671286 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-ovn\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671301 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-kubernetes\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671241 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671345 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-kubernetes\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671343 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-system-cni-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671408 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-netd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671423 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-device-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.671439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671439 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-system-cni-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671466 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-netd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671518 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-systemd\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671552 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-host\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671567 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-kubelet\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671573 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-systemd\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671580 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-log-socket\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671604 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-device-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671612 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-hosts-file\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671642 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-kubelet\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671650 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-log-socket\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671651 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-kubelet\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671652 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-host\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671660 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-hosts-file\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671689 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-kubelet\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671710 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-socket-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671755 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-socket-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671778 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671804 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-run\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671868 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671890 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-lib-modules\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671926 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-cnibin\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671997 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-lib-modules\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.672022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.671989 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-node-log\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672036 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-cnibin\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672053 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-bin\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672114 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-node-log\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672118 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672159 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-cni-bin\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672162 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-registration-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672197 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-sys\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672243 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672296 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-var-lib-kubelet\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672315 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-sys\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672347 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ace2d1f7-e48a-4375-848d-4cd0f9122046-crio-socket\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672360 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672364 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-run\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672415 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-netns\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672432 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-var-lib-kubelet\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672452 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ace2d1f7-e48a-4375-848d-4cd0f9122046-crio-socket\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672443 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-etc-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672476 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-etc-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672475 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-run-netns\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672501 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-hostroot\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672519 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-netns\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672536 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-hostroot\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672533 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a69377-a3ea-42ab-8a85-753770827b97-host\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672565 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-netns\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672563 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672591 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a69377-a3ea-42ab-8a85-753770827b97-host\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672611 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-systemd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672627 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-systemd-units\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672653 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-systemd-units\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672666 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-systemd\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672695 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-registration-dir\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672705 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-conf\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672762 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-kubelet-config\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672789 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-multus-certs\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672817 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-var-lib-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672841 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cnibin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672856 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/689471e9-1161-4ae7-b08a-15b85ffaa359-kubelet-config\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672863 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysctl-conf\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672864 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-k8s-cni-cncf-io\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672858 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-multus-certs\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672890 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-var-lib-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672897 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-multus\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672916 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-run-k8s-cni-cncf-io\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672926 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-slash\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672903 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-cnibin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672943 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysconfig\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672965 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672976 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-host-slash\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672990 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-bin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.672996 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-sysconfig\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673005 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bade2e15-5673-4f0e-9fa4-fec7d544507c-run-openvswitch\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673019 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-sys-fs\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673040 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-bin\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673043 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.673018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673043 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-host-var-lib-cni-multus\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673069 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673094 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65662b67-e317-41c9-959e-24c40741d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673112 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9d566de-6bf8-4fc4-966b-ac984e4a300e-sys-fs\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673113 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-etc-kubernetes\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673135 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-etc-kubernetes\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673138 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4220aacf-5c03-4983-904f-6b0d522cb8c6-host-slash\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673155 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-modprobe-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673170 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-system-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673183 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673197 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4220aacf-5c03-4983-904f-6b0d522cb8c6-host-slash\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673202 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-socket-dir-parent\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673229 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-socket-dir-parent\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673232 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-system-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673247 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-cni-dir\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.674585 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.673279 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d633766-ee45-4299-a89f-7db3cd012d43-etc-modprobe-d\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.711181 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711119 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.711265 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711225 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.711265 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711258 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.711365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711281 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.711365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711293 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.711365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711293 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:30:47.711365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711337 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.711494 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711388 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.711648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711620 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.711648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711628 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:30:47.711799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711650 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:30:47.711799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711674 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:30:47.711799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711684 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.711799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711751 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.711799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711776 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:30:47.712000 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711811 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.712000 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711866 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.712000 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711828 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-852xh\" (UniqueName: \"kubernetes.io/projected/65662b67-e317-41c9-959e-24c40741d793-kube-api-access-852xh\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.712000 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711931 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.712000 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.711966 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.712222 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712149 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.712222 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712181 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.712222 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712203 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.712222 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712204 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:30:47.712352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712338 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:30:47.712394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712379 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.712571 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712508 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:30:47.712571 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712516 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 20:30:47.712571 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712535 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:30:47.712571 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712550 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:30:47.712571 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712548 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:30:47.712979 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712959 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbhw\" (UniqueName: \"kubernetes.io/projected/f62b13d3-9a85-404d-b7c5-0a482d942bf3-kube-api-access-9tbhw\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.713035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.712991 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:30:47.713328 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.713312 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27cw\" (UniqueName: \"kubernetes.io/projected/e9d566de-6bf8-4fc4-966b-ac984e4a300e-kube-api-access-b27cw\") pod \"aws-ebs-csi-driver-node-xxn4k\" (UID: \"e9d566de-6bf8-4fc4-966b-ac984e4a300e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxn4k" Nov 01 20:30:47.713526 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.713503 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmx8q\" (UniqueName: \"kubernetes.io/projected/2e69a8f9-67ff-4ee6-9718-ab2c186021a1-kube-api-access-bmx8q\") pod \"node-resolver-2lphr\" (UID: \"2e69a8f9-67ff-4ee6-9718-ab2c186021a1\") " pod="openshift-dns/node-resolver-2lphr" Nov 01 20:30:47.713888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.713872 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9h84\" (UniqueName: \"kubernetes.io/projected/04878621-4728-4a68-8e75-1af069bec931-kube-api-access-m9h84\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:47.715710 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.715696 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrpr\" (UniqueName: \"kubernetes.io/projected/1544a0e3-068d-4d70-bbfb-df3e4f86f391-kube-api-access-bhrpr\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:47.715814 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.715797 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznxf\" (UniqueName: \"kubernetes.io/projected/0c893f01-c22e-4e7d-a4ce-8e7896ef0e09-kube-api-access-dznxf\") pod \"migrator-66df5d496b-kshpn\" (UID: \"0c893f01-c22e-4e7d-a4ce-8e7896ef0e09\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kshpn" Nov 01 20:30:47.715846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.715799 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mqs\" (UniqueName: \"kubernetes.io/projected/6d9a1fb0-d057-4922-97a5-62ea9bebf833-kube-api-access-j5mqs\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.715846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.715799 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlt4b\" (UniqueName: \"kubernetes.io/projected/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-api-access-nlt4b\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.716000 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.715987 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqbjz\" (UniqueName: \"kubernetes.io/projected/96533abd-b0c9-46eb-bf51-96e4c58da84d-kube-api-access-fqbjz\") pod \"network-check-target-sgmhx\" (UID: \"96533abd-b0c9-46eb-bf51-96e4c58da84d\") " pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:47.718706 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.718691 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.718798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.718785 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-config\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.718898 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.718883 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:47.718928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.718913 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f62b13d3-9a85-404d-b7c5-0a482d942bf3-multus-daemon-config\") pod \"multus-cq22n\" (UID: \"f62b13d3-9a85-404d-b7c5-0a482d942bf3\") " pod="openshift-multus/multus-cq22n" Nov 01 20:30:47.719025 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.719013 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ace2d1f7-e48a-4375-848d-4cd0f9122046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:47.719855 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.719840 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65662b67-e317-41c9-959e-24c40741d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zp6f6\" (UID: \"65662b67-e317-41c9-959e-24c40741d793\") " pod="openshift-multus/multus-additional-cni-plugins-zp6f6" Nov 01 20:30:47.719907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.719854 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1fb0-d057-4922-97a5-62ea9bebf833-config-volume\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.721192 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.720978 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-env-overrides\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.721192 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.721144 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4220aacf-5c03-4983-904f-6b0d522cb8c6-iptables-alerter-script\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:47.721334 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.721275 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovnkube-script-lib\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.721595 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.721561 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6v8m\" (UniqueName: \"kubernetes.io/projected/bade2e15-5673-4f0e-9fa4-fec7d544507c-kube-api-access-h6v8m\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.721839 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.721820 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40a69377-a3ea-42ab-8a85-753770827b97-serviceca\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:47.721913 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.721823 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64fm9\" (UniqueName: \"kubernetes.io/projected/4220aacf-5c03-4983-904f-6b0d522cb8c6-kube-api-access-64fm9\") pod \"iptables-alerter-kdh6b\" (UID: \"4220aacf-5c03-4983-904f-6b0d522cb8c6\") " pod="openshift-network-operator/iptables-alerter-kdh6b" Nov 01 20:30:47.722016 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.721993 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcjb6\" (UniqueName: \"kubernetes.io/projected/40a69377-a3ea-42ab-8a85-753770827b97-kube-api-access-tcjb6\") pod \"node-ca-nrcnn\" (UID: \"40a69377-a3ea-42ab-8a85-753770827b97\") " pod="openshift-image-registry/node-ca-nrcnn" Nov 01 20:30:47.722051 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.722002 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9z69\" (UniqueName: \"kubernetes.io/projected/9d633766-ee45-4299-a89f-7db3cd012d43-kube-api-access-s9z69\") pod \"tuned-mqgcd\" (UID: \"9d633766-ee45-4299-a89f-7db3cd012d43\") " pod="openshift-cluster-node-tuning-operator/tuned-mqgcd" Nov 01 20:30:47.723140 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.723106 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 20:30:47.723248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.723138 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxvs\" (UniqueName: \"kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:47.724354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.723322 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:30:47.728672 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.728654 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jgj4\"" Nov 01 20:30:47.730925 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.730902 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle\") pod \"periodic-gathering-8mzmm-bpj4x\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:30:47.732946 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:47.732932 4477 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:47.733021 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:47.733009 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:48.232988337 +0000 UTC m=+1.883039887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:47.760864 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.760822 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t8z8c\"" Nov 01 20:30:47.761104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.760998 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pskvw\"" Nov 01 20:30:47.761104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761014 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9m2k9\"" Nov 01 20:30:47.761104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761030 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:30:47.761311 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761293 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j72sw\"" Nov 01 20:30:47.761381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761306 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4gdtb\"" Nov 01 20:30:47.761381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761335 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5cgzw\"" Nov 01 20:30:47.761499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761390 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:30:47.761499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761419 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 20:30:47.761499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761462 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r4cnd\"" Nov 01 20:30:47.761499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761467 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cg54x\"" Nov 01 20:30:47.761499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761466 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:30:47.761687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761474 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n2j5c\"" Nov 01 20:30:47.761687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761521 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:30:47.761687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761579 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ggf7k\"" Nov 01 20:30:47.761687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761630 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kzbx4\"" Nov 01 20:30:47.761809 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761763 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:30:47.761849 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.761816 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wk8gn\"" Nov 01 20:30:47.763192 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.763175 4477 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 20:30:47.765357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.765336 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d9a1fb0-d057-4922-97a5-62ea9bebf833-metrics-tls\") pod \"dns-default-w9xfp\" (UID: \"6d9a1fb0-d057-4922-97a5-62ea9bebf833\") " pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:47.765449 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.765430 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1544a0e3-068d-4d70-bbfb-df3e4f86f391-metrics-certs\") pod \"network-metrics-daemon-mtclz\" (UID: \"1544a0e3-068d-4d70-bbfb-df3e4f86f391\") " pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:47.770582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.770560 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04878621-4728-4a68-8e75-1af069bec931-cert\") pod \"ingress-canary-bkh8x\" (UID: \"04878621-4728-4a68-8e75-1af069bec931\") " pod="openshift-ingress-canary/ingress-canary-bkh8x" Nov 01 20:30:47.770722 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.770701 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/689471e9-1161-4ae7-b08a-15b85ffaa359-original-pull-secret\") pod \"global-pull-secret-syncer-658ds\" (UID: \"689471e9-1161-4ae7-b08a-15b85ffaa359\") " pod="kube-system/global-pull-secret-syncer-658ds" Nov 01 20:30:47.771680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.771665 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bade2e15-5673-4f0e-9fa4-fec7d544507c-ovn-node-metrics-cert\") pod \"ovnkube-node-hrnnn\" (UID: \"bade2e15-5673-4f0e-9fa4-fec7d544507c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:47.861835 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.861810 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtclz" Nov 01 20:30:47.862137 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:47.862123 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:48.254853 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:48.254736 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mtclz"] Nov 01 20:30:48.258094 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:48.258050 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1544a0e3_068d_4d70_bbfb_df3e4f86f391.slice/crio-01a7c3e3921f6b896cbc8780794470f739f88dc44201b0e44dc8ddea6ad44fad WatchSource:0}: Error finding container 01a7c3e3921f6b896cbc8780794470f739f88dc44201b0e44dc8ddea6ad44fad: Status 404 returned error can't find the container with id 01a7c3e3921f6b896cbc8780794470f739f88dc44201b0e44dc8ddea6ad44fad Nov 01 20:30:48.260282 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:48.260267 4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:30:48.278502 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:48.278482 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:48.278603 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:48.278589 4477 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:48.278658 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:48.278651 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:49.278634182 +0000 UTC m=+2.928685707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:48.305034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:48.305007 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w9xfp"] Nov 01 20:30:48.311686 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:48.311664 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9a1fb0_d057_4922_97a5_62ea9bebf833.slice/crio-2473ecb24d462b723c34c781a0c0ee044e4c3401bfb99bb6002885f7e8e9af46 WatchSource:0}: Error finding container 2473ecb24d462b723c34c781a0c0ee044e4c3401bfb99bb6002885f7e8e9af46: Status 404 returned error can't find the container with id 2473ecb24d462b723c34c781a0c0ee044e4c3401bfb99bb6002885f7e8e9af46 Nov 01 20:30:48.362064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:48.362037 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrcnn_40a69377-a3ea-42ab-8a85-753770827b97/node-ca/0.log" Nov 01 20:30:48.537456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:48.537386 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9xfp" event={"ID":"6d9a1fb0-d057-4922-97a5-62ea9bebf833","Type":"ContainerStarted","Data":"2473ecb24d462b723c34c781a0c0ee044e4c3401bfb99bb6002885f7e8e9af46"} Nov 01 20:30:48.538540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:48.538518 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtclz" event={"ID":"1544a0e3-068d-4d70-bbfb-df3e4f86f391","Type":"ContainerStarted","Data":"01a7c3e3921f6b896cbc8780794470f739f88dc44201b0e44dc8ddea6ad44fad"} Nov 01 20:30:48.963342 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:48.963313 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bkh8x_04878621-4728-4a68-8e75-1af069bec931/serve-healthcheck-canary/0.log" Nov 01 20:30:49.226034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:49.225944 4477 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9msm2_c481a350-2dc5-450b-b59c-2a37640d458e/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:49.284202 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:49.284168 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:49.284391 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:49.284312 4477 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:49.284391 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:30:49.284379 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls podName:ace2d1f7-e48a-4375-848d-4cd0f9122046 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:51.284359779 +0000 UTC m=+4.934411303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xtldx" (UID: "ace2d1f7-e48a-4375-848d-4cd0f9122046") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:49.639749 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:49.639712 4477 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9msm2_c481a350-2dc5-450b-b59c-2a37640d458e/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:49.792251 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:49.792220 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-kshpn_0c893f01-c22e-4e7d-a4ce-8e7896ef0e09/migrator/0.log" Nov 01 20:30:49.845295 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:49.845245 4477 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9msm2_c481a350-2dc5-450b-b59c-2a37640d458e/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:50.051270 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:50.051199 4477 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9msm2_c481a350-2dc5-450b-b59c-2a37640d458e/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:50.545853 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:50.545823 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtclz" event={"ID":"1544a0e3-068d-4d70-bbfb-df3e4f86f391","Type":"ContainerStarted","Data":"9b1031a450309326b1446a33bb74b8f208729f5df1a018a760c5062138877be2"} Nov 01 20:30:50.546026 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:50.545873 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtclz" event={"ID":"1544a0e3-068d-4d70-bbfb-df3e4f86f391","Type":"ContainerStarted","Data":"928bff858b8d909f4e273ef8a49b8544f211dd053e3615ce12ce14317262f102"} Nov 01 20:30:51.301041 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:51.301004 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:51.303661 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:51.303631 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ace2d1f7-e48a-4375-848d-4cd0f9122046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xtldx\" (UID: \"ace2d1f7-e48a-4375-848d-4cd0f9122046\") " pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:51.463389 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:51.463357 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xtldx" Nov 01 20:30:51.552876 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:51.552808 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9xfp" event={"ID":"6d9a1fb0-d057-4922-97a5-62ea9bebf833","Type":"ContainerStarted","Data":"d9696a7c869c8e8cfcfbe6795dfc82e1d96edcc226c58b9a1f0021b99dba4a9f"} Nov 01 20:30:51.552876 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:51.552864 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9xfp" event={"ID":"6d9a1fb0-d057-4922-97a5-62ea9bebf833","Type":"ContainerStarted","Data":"71f352b13fcb3eefe86367c18068762473731b2962ef8c218ca07636e81d0aaf"} Nov 01 20:30:51.825927 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:51.825890 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xtldx"] Nov 01 20:30:51.830234 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:30:51.830206 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace2d1f7_e48a_4375_848d_4cd0f9122046.slice/crio-341c1828d417d03c401f507b90e12e4689211a11469dcf5a2483616da4111bc2 WatchSource:0}: Error finding container 341c1828d417d03c401f507b90e12e4689211a11469dcf5a2483616da4111bc2: Status 404 returned error can't find the container with id 341c1828d417d03c401f507b90e12e4689211a11469dcf5a2483616da4111bc2 Nov 01 20:30:52.559846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:52.559812 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xtldx" event={"ID":"ace2d1f7-e48a-4375-848d-4cd0f9122046","Type":"ContainerStarted","Data":"bb0c5057859b3f2d7f6f60570dac2f69c62d25b79f1cb2ea7bad09595c26a7e8"} Nov 01 20:30:52.560244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:52.559889 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xtldx" event={"ID":"ace2d1f7-e48a-4375-848d-4cd0f9122046","Type":"ContainerStarted","Data":"341c1828d417d03c401f507b90e12e4689211a11469dcf5a2483616da4111bc2"} Nov 01 20:30:52.560244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:52.559997 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w9xfp" Nov 01 20:30:52.860966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:52.860934 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:53.565034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:53.564998 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xtldx" event={"ID":"ace2d1f7-e48a-4375-848d-4cd0f9122046","Type":"ContainerStarted","Data":"a923eced31b9f6f39fb65b09ce0adac708842472ec9a7c78b997c8a4d71315bb"} Nov 01 20:30:53.952835 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:53.952750 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:30:55.575731 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:55.575696 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xtldx" event={"ID":"ace2d1f7-e48a-4375-848d-4cd0f9122046","Type":"ContainerStarted","Data":"f12e561a016594a3c96f2fa827955f1befde2d52fd5074f59e5c258943e3cc9b"} Nov 01 20:30:57.331936 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:57.331904 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:30:57.332328 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:30:57.332165 4477 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:31:02.567158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:02.567127 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w9xfp" Nov 01 20:31:06.551841 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:31:06.551794 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:31:07.044830 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:07.044796 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrnnn" Nov 01 20:31:09.213855 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.213822 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9msm2" Nov 01 20:31:09.215264 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.215243 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9cfd6f878-mzshm"] Nov 01 20:31:09.216998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.216984 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.314400 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.314358 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-serving-cert\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.314589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.314430 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-oauth-serving-cert\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.314589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.314575 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-config\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.314664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.314610 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37cf081a-11c9-46ce-abb0-e4d20050d25f-tmp\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.314664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.314641 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprjr\" (UniqueName: \"kubernetes.io/projected/37cf081a-11c9-46ce-abb0-e4d20050d25f-kube-api-access-pprjr\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.314756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.314677 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-oauth-config\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.314806 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.314791 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-service-ca\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.319324 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.319290 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 20:31:09.319324 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.319310 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 20:31:09.319324 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.319296 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bxzf5\"" Nov 01 20:31:09.319643 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.319628 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 20:31:09.319690 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.319628 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 20:31:09.319690 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.319635 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 20:31:09.320015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.319997 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9cfd6f878-mzshm"] Nov 01 20:31:09.368926 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.368890 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 20:31:09.369131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.368959 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 20:31:09.415496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.415464 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-serving-cert\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.415496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.415498 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-oauth-serving-cert\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.415680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.415518 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-config\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.415680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.415538 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37cf081a-11c9-46ce-abb0-e4d20050d25f-tmp\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.415680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.415556 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprjr\" (UniqueName: \"kubernetes.io/projected/37cf081a-11c9-46ce-abb0-e4d20050d25f-kube-api-access-pprjr\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.415680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.415576 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-oauth-config\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.415680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.415596 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-service-ca\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.416025 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.415998 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37cf081a-11c9-46ce-abb0-e4d20050d25f-tmp\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.416446 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.416428 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-service-ca\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.416514 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.416444 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-oauth-serving-cert\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.416514 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.416444 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-config\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.418601 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.418571 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-oauth-config\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.418884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.418866 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-serving-cert\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.477371 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.477298 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprjr\" (UniqueName: \"kubernetes.io/projected/37cf081a-11c9-46ce-abb0-e4d20050d25f-kube-api-access-pprjr\") pod \"console-9cfd6f878-mzshm\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.526359 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.526321 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:09.890744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.890714 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9cfd6f878-mzshm"] Nov 01 20:31:09.925009 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:31:09.924980 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cf081a_11c9_46ce_abb0_e4d20050d25f.slice/crio-ddb525def1c4f45e581d5c47cfd25f1961f179fc68d63ebc3dd9eccd876bfdfb WatchSource:0}: Error finding container ddb525def1c4f45e581d5c47cfd25f1961f179fc68d63ebc3dd9eccd876bfdfb: Status 404 returned error can't find the container with id ddb525def1c4f45e581d5c47cfd25f1961f179fc68d63ebc3dd9eccd876bfdfb Nov 01 20:31:09.931429 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.931405 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv"] Nov 01 20:31:09.933686 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.933671 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:09.988519 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.988482 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 20:31:09.988751 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.988661 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 20:31:09.988808 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.988783 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 20:31:09.988808 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.988783 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4xs6x\"" Nov 01 20:31:09.988884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:09.988783 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 20:31:10.019312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.019283 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/437a2f7a-e562-480f-a73c-5c8ea695a375-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.019312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.019314 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jrbw\" (UniqueName: \"kubernetes.io/projected/437a2f7a-e562-480f-a73c-5c8ea695a375-kube-api-access-8jrbw\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.019509 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.019333 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/437a2f7a-e562-480f-a73c-5c8ea695a375-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.019509 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.019353 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/437a2f7a-e562-480f-a73c-5c8ea695a375-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.031575 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.031552 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv"] Nov 01 20:31:10.089484 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.089450 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 20:31:10.119985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.119953 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/437a2f7a-e562-480f-a73c-5c8ea695a375-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.119985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.119987 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jrbw\" (UniqueName: \"kubernetes.io/projected/437a2f7a-e562-480f-a73c-5c8ea695a375-kube-api-access-8jrbw\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.120213 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.120010 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/437a2f7a-e562-480f-a73c-5c8ea695a375-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.120213 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.120031 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/437a2f7a-e562-480f-a73c-5c8ea695a375-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.120799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.120761 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/437a2f7a-e562-480f-a73c-5c8ea695a375-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.122586 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.122564 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/437a2f7a-e562-480f-a73c-5c8ea695a375-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.122663 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.122585 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/437a2f7a-e562-480f-a73c-5c8ea695a375-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.176709 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.176639 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jrbw\" (UniqueName: \"kubernetes.io/projected/437a2f7a-e562-480f-a73c-5c8ea695a375-kube-api-access-8jrbw\") pod \"prometheus-operator-9cd89b74c-tn6bv\" (UID: \"437a2f7a-e562-480f-a73c-5c8ea695a375\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.244672 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.244641 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" Nov 01 20:31:10.605691 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.605660 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv"] Nov 01 20:31:10.608613 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:31:10.608581 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437a2f7a_e562_480f_a73c_5c8ea695a375.slice/crio-a8f54957bb19ba5f6630236c414c3c5066421fc06d94b37522307368b60da97a WatchSource:0}: Error finding container a8f54957bb19ba5f6630236c414c3c5066421fc06d94b37522307368b60da97a: Status 404 returned error can't find the container with id a8f54957bb19ba5f6630236c414c3c5066421fc06d94b37522307368b60da97a Nov 01 20:31:10.622440 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.622411 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" event={"ID":"437a2f7a-e562-480f-a73c-5c8ea695a375","Type":"ContainerStarted","Data":"a8f54957bb19ba5f6630236c414c3c5066421fc06d94b37522307368b60da97a"} Nov 01 20:31:10.623539 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:10.623519 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cfd6f878-mzshm" event={"ID":"37cf081a-11c9-46ce-abb0-e4d20050d25f","Type":"ContainerStarted","Data":"ddb525def1c4f45e581d5c47cfd25f1961f179fc68d63ebc3dd9eccd876bfdfb"} Nov 01 20:31:12.633188 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:12.633124 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" event={"ID":"437a2f7a-e562-480f-a73c-5c8ea695a375","Type":"ContainerStarted","Data":"60476a42bec16b1fc169998bdf7aaf1bca779bf26d2719420eb97a3de5e484fe"} Nov 01 20:31:13.638131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:13.638093 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" event={"ID":"437a2f7a-e562-480f-a73c-5c8ea695a375","Type":"ContainerStarted","Data":"c95221fd2dbe04401f97fc302825b4eaeca4b9dea8fffc7bb76f465528d7cd1a"} Nov 01 20:31:13.639751 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:13.639723 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cfd6f878-mzshm" event={"ID":"37cf081a-11c9-46ce-abb0-e4d20050d25f","Type":"ContainerStarted","Data":"78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27"} Nov 01 20:31:13.751557 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:13.751475 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tn6bv" podStartSLOduration=2.793935537 podStartE2EDuration="4.75146174s" totalImagesPullingTime="1.957526203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:13.750452019 +0000 UTC m=+27.400503588" watchObservedRunningTime="2026-11-01 20:31:13.75146174 +0000 UTC m=+27.401513284" Nov 01 20:31:13.864437 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:13.864381 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9cfd6f878-mzshm" podStartSLOduration=1.396845701 podStartE2EDuration="4.864353279s" totalImagesPullingTime="3.467507578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:13.86389344 +0000 UTC m=+27.513945007" watchObservedRunningTime="2026-11-01 20:31:13.864353279 +0000 UTC m=+27.514404817" Nov 01 20:31:16.532606 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.532574 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j9v7z"] Nov 01 20:31:16.536812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.536791 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.567859 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.567824 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-textfile\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.568022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.567864 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-wtmp\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.568022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.567894 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.568144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.568034 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1462938c-0e23-476c-acf3-9a7e13acfc99-metrics-client-ca\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.568144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.568065 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-tls\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.568144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.568132 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-root\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.568248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.568153 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg78t\" (UniqueName: \"kubernetes.io/projected/1462938c-0e23-476c-acf3-9a7e13acfc99-kube-api-access-fg78t\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.568248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.568221 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-sys\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.568248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.568242 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-accelerators-collector-config\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.582290 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.582262 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn"] Nov 01 20:31:16.585294 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.585273 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.631281 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.631252 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 20:31:16.631281 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.631270 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rcjlq\"" Nov 01 20:31:16.631281 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.631267 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 20:31:16.631506 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.631257 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 20:31:16.640127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.640095 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 20:31:16.640127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.640106 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 20:31:16.640483 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.640460 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z5gjb\"" Nov 01 20:31:16.641182 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.641161 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn"] Nov 01 20:31:16.669344 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669319 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.669476 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669358 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-sys\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669476 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669381 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-accelerators-collector-config\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669476 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669405 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5msk\" (UniqueName: \"kubernetes.io/projected/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-api-access-m5msk\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.669476 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669423 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-textfile\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669605 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669482 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-wtmp\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669605 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669497 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-sys\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669605 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669563 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.669744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669626 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669640 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-wtmp\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669702 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1462938c-0e23-476c-acf3-9a7e13acfc99-metrics-client-ca\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669744 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-tls\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669773 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-root\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669799 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg78t\" (UniqueName: \"kubernetes.io/projected/1462938c-0e23-476c-acf3-9a7e13acfc99-kube-api-access-fg78t\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.669891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669823 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.669891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669861 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.670126 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669889 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.670126 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669900 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-textfile\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.670126 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.669922 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1462938c-0e23-476c-acf3-9a7e13acfc99-root\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.670268 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.670169 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-accelerators-collector-config\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.670309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.670291 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1462938c-0e23-476c-acf3-9a7e13acfc99-metrics-client-ca\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.672298 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.672278 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-tls\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.672609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.672592 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1462938c-0e23-476c-acf3-9a7e13acfc99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.706207 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.706190 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 20:31:16.726895 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.726874 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg78t\" (UniqueName: \"kubernetes.io/projected/1462938c-0e23-476c-acf3-9a7e13acfc99-kube-api-access-fg78t\") pod \"node-exporter-j9v7z\" (UID: \"1462938c-0e23-476c-acf3-9a7e13acfc99\") " pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.771271 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.771246 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.771378 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.771297 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.771378 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.771330 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.771378 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.771347 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.771378 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.771368 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.771584 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.771408 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5msk\" (UniqueName: \"kubernetes.io/projected/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-api-access-m5msk\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.771789 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.771768 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.771949 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.771928 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.772120 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.772100 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.773750 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.773730 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.773824 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.773806 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.832643 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.832623 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5msk\" (UniqueName: \"kubernetes.io/projected/2a97282f-8f28-4dfe-9c5d-1fb7a25c9484-kube-api-access-m5msk\") pod \"kube-state-metrics-5dc8d7d89b-ggqkn\" (UID: \"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:16.852589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.852570 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j9v7z" Nov 01 20:31:16.872973 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:31:16.872953 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1462938c_0e23_476c_acf3_9a7e13acfc99.slice/crio-f231e392980870f28b7086f6de5db9e7f0af1354963ceaee605ec9c0869fa2e5 WatchSource:0}: Error finding container f231e392980870f28b7086f6de5db9e7f0af1354963ceaee605ec9c0869fa2e5: Status 404 returned error can't find the container with id f231e392980870f28b7086f6de5db9e7f0af1354963ceaee605ec9c0869fa2e5 Nov 01 20:31:16.902927 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:16.902908 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" Nov 01 20:31:17.336559 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:17.336521 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn"] Nov 01 20:31:17.340194 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:31:17.340168 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a97282f_8f28_4dfe_9c5d_1fb7a25c9484.slice/crio-eb3dfa29678c71443b80aa0fef4e9a5a6fbfcd6334f3add20c3fca2315c77ce8 WatchSource:0}: Error finding container eb3dfa29678c71443b80aa0fef4e9a5a6fbfcd6334f3add20c3fca2315c77ce8: Status 404 returned error can't find the container with id eb3dfa29678c71443b80aa0fef4e9a5a6fbfcd6334f3add20c3fca2315c77ce8 Nov 01 20:31:17.655300 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:17.655214 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" event={"ID":"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484","Type":"ContainerStarted","Data":"eb3dfa29678c71443b80aa0fef4e9a5a6fbfcd6334f3add20c3fca2315c77ce8"} Nov 01 20:31:17.656407 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:17.656385 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j9v7z" event={"ID":"1462938c-0e23-476c-acf3-9a7e13acfc99","Type":"ContainerStarted","Data":"f231e392980870f28b7086f6de5db9e7f0af1354963ceaee605ec9c0869fa2e5"} Nov 01 20:31:18.857368 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.857343 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:31:18.874955 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.874935 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.917716 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.917688 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:31:18.928546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.928517 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 20:31:18.928546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.928531 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 20:31:18.928744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.928554 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 20:31:18.929107 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.929066 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 20:31:18.961626 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.961580 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 20:31:18.965027 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.964998 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 20:31:18.965173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.965068 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 20:31:18.965173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.965106 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 20:31:18.965173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.965148 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w8s42\"" Nov 01 20:31:18.988439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988410 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988597 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988461 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988597 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988483 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988597 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988560 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9k2l\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-kube-api-access-t9k2l\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988610 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988674 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988698 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988823 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988730 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988812 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-config-out\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988853 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988925 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988871 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-web-config\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988925 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988897 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:18.988990 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:18.988950 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.028117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.028013 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 20:31:19.089733 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089703 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.089928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089740 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.089928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089774 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.089928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089808 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-config-out\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.089928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089829 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.089928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089843 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-web-config\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.089928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089865 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.089928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089894 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.089928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089922 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.090347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089946 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.090347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089973 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.090347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.089998 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9k2l\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-kube-api-access-t9k2l\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.090347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.090020 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.090571 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.090420 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.090627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.090582 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.091966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.091938 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.093018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.092995 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.093248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.093227 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.093816 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.093612 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.093816 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.093779 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.093963 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.093835 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-config-out\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.093963 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.093922 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.094144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.094118 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.095109 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.095062 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-web-config\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.095302 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.095280 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.151865 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.151830 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9k2l\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-kube-api-access-t9k2l\") pod \"alertmanager-main-0\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.268844 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.268810 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:31:19.527397 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.527360 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:19.527587 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.527426 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:19.533141 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.533111 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:19.666342 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.666308 4477 generic.go:356] "Generic (PLEG): container finished" podID="1462938c-0e23-476c-acf3-9a7e13acfc99" containerID="6e80dbe2c6696e62c5ef028f40c933e1856ca102e6f5565491128eef58375a9d" exitCode=0 Nov 01 20:31:19.666539 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.666394 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j9v7z" event={"ID":"1462938c-0e23-476c-acf3-9a7e13acfc99","Type":"ContainerDied","Data":"6e80dbe2c6696e62c5ef028f40c933e1856ca102e6f5565491128eef58375a9d"} Nov 01 20:31:19.670561 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.670541 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:31:19.772568 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:19.772543 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:31:19.873482 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:31:19.873452 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9199082_eca3_44a0_8a29_5398460a1ca7.slice/crio-bb3935bb27d7f6fede1e7eff2d102ae124156878df2627173ed5e15e8a33e87d WatchSource:0}: Error finding container bb3935bb27d7f6fede1e7eff2d102ae124156878df2627173ed5e15e8a33e87d: Status 404 returned error can't find the container with id bb3935bb27d7f6fede1e7eff2d102ae124156878df2627173ed5e15e8a33e87d Nov 01 20:31:20.672297 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.672260 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerStarted","Data":"bb3935bb27d7f6fede1e7eff2d102ae124156878df2627173ed5e15e8a33e87d"} Nov 01 20:31:20.675379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.674602 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" event={"ID":"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484","Type":"ContainerStarted","Data":"e1c0fd0019eee3c0083aff27947fdc0105a902d62a937c14a157920eda9ecb63"} Nov 01 20:31:20.675379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.674684 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" event={"ID":"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484","Type":"ContainerStarted","Data":"990b4a5df3f217abdb00c2c17a4be8f344c2e28d2e63b79f491bed04115745a6"} Nov 01 20:31:20.675379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.674711 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" event={"ID":"2a97282f-8f28-4dfe-9c5d-1fb7a25c9484","Type":"ContainerStarted","Data":"1459a64a2436e7686aea6dd46f83fbdf033c98dd73adbfcaad1dc414a22942e5"} Nov 01 20:31:20.677139 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.677116 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j9v7z" event={"ID":"1462938c-0e23-476c-acf3-9a7e13acfc99","Type":"ContainerStarted","Data":"0af8a91a03696aba1fc173cdb8621a66ecbae77694d8266551ac89546b8c3c85"} Nov 01 20:31:20.677258 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.677158 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j9v7z" event={"ID":"1462938c-0e23-476c-acf3-9a7e13acfc99","Type":"ContainerStarted","Data":"60d603604040d58a1477308d451bc7e2fcf2403b634f4e0e715dda25a7eef0c4"} Nov 01 20:31:20.797217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.797128 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ggqkn" podStartSLOduration=2.240275407 podStartE2EDuration="4.797111576s" totalImagesPullingTime="2.556836169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:20.796739304 +0000 UTC m=+34.446790868" watchObservedRunningTime="2026-11-01 20:31:20.797111576 +0000 UTC m=+34.447163117" Nov 01 20:31:20.914377 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.914317 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j9v7z" podStartSLOduration=2.93271252 podStartE2EDuration="4.91430213s" totalImagesPullingTime="1.88487119s" totalInitContainerRuntime="96.71842ms" isStatefulPod=true podCreationTimestamp="2026-11-01 20:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:20.913195084 +0000 UTC m=+34.563246632" watchObservedRunningTime="2026-11-01 20:31:20.91430213 +0000 UTC m=+34.564353688" Nov 01 20:31:20.948363 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.948305 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c8495845b-9xnpv"] Nov 01 20:31:20.965639 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:20.965614 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.008769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.008736 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-trusted-ca-bundle\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.008769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.008772 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tn9\" (UniqueName: \"kubernetes.io/projected/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-kube-api-access-k5tn9\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.008975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.008865 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-tmp\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.008975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.008910 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-service-ca\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.008975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.008940 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-oauth-serving-cert\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.009104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.009006 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-config\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.009104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.009026 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-serving-cert\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.009104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.009050 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-oauth-config\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.010945 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.010920 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8495845b-9xnpv"] Nov 01 20:31:21.067944 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.067908 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 20:31:21.110665 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.110630 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-trusted-ca-bundle\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.110832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.110680 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tn9\" (UniqueName: \"kubernetes.io/projected/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-kube-api-access-k5tn9\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.110832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.110732 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-tmp\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.110832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.110776 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-service-ca\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.110832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.110829 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-oauth-serving-cert\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.111066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.110874 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-config\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.111066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.110904 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-serving-cert\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.111066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.110943 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-oauth-config\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.111942 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.111913 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-trusted-ca-bundle\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.112414 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.112391 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-oauth-serving-cert\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.112496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.112457 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-service-ca\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.112645 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.112620 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-tmp\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.112963 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.112941 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-config\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.113866 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.113842 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-oauth-config\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.114533 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.114509 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-serving-cert\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.168364 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.168334 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tn9\" (UniqueName: \"kubernetes.io/projected/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-kube-api-access-k5tn9\") pod \"console-c8495845b-9xnpv\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.288279 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.288195 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:21.670754 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.670719 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8495845b-9xnpv"] Nov 01 20:31:21.673963 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:31:21.673935 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b6ce0b_61ea_4525_a727_99b1ada7f62d.slice/crio-fae3d46714e8d9a9c7ed35e6728c9250a551f2aa7e2b50949aa383c3d3e52664 WatchSource:0}: Error finding container fae3d46714e8d9a9c7ed35e6728c9250a551f2aa7e2b50949aa383c3d3e52664: Status 404 returned error can't find the container with id fae3d46714e8d9a9c7ed35e6728c9250a551f2aa7e2b50949aa383c3d3e52664 Nov 01 20:31:21.681302 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:21.681276 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8495845b-9xnpv" event={"ID":"f5b6ce0b-61ea-4525-a727-99b1ada7f62d","Type":"ContainerStarted","Data":"fae3d46714e8d9a9c7ed35e6728c9250a551f2aa7e2b50949aa383c3d3e52664"} Nov 01 20:31:22.686210 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:22.686165 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerID="efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417" exitCode=0 Nov 01 20:31:22.686684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:22.686242 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerDied","Data":"efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417"} Nov 01 20:31:22.688008 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:22.687985 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8495845b-9xnpv" event={"ID":"f5b6ce0b-61ea-4525-a727-99b1ada7f62d","Type":"ContainerStarted","Data":"6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a"} Nov 01 20:31:22.931406 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:22.931356 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c8495845b-9xnpv" podStartSLOduration=2.931341577 podStartE2EDuration="2.931341577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:22.930095088 +0000 UTC m=+36.580146649" watchObservedRunningTime="2026-11-01 20:31:22.931341577 +0000 UTC m=+36.581393123" Nov 01 20:31:23.608714 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.608679 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c8495845b-9xnpv"] Nov 01 20:31:23.647409 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.647376 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ccf46c54d-zg952"] Nov 01 20:31:23.649861 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.649845 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.711125 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.711069 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ccf46c54d-zg952"] Nov 01 20:31:23.733987 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.733960 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dbba653-30e6-4e49-94a3-f666b4409c89-tmp\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.734134 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.733995 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw65f\" (UniqueName: \"kubernetes.io/projected/3dbba653-30e6-4e49-94a3-f666b4409c89-kube-api-access-zw65f\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.734134 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.734027 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-oauth-serving-cert\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.734218 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.734144 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-console-config\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.734218 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.734208 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-service-ca\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.734278 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.734228 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-oauth-config\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.734307 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.734277 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-serving-cert\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.734307 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.734295 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-trusted-ca-bundle\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835199 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-service-ca\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835239 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-oauth-config\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835460 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835263 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-serving-cert\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835460 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835303 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-trusted-ca-bundle\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835460 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835388 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dbba653-30e6-4e49-94a3-f666b4409c89-tmp\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835460 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835427 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw65f\" (UniqueName: \"kubernetes.io/projected/3dbba653-30e6-4e49-94a3-f666b4409c89-kube-api-access-zw65f\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835524 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-oauth-serving-cert\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835556 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-console-config\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.835987 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.835962 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-service-ca\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.836288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.836266 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-console-config\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.836354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.836328 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-trusted-ca-bundle\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.836406 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.836383 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-oauth-serving-cert\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.836560 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.836539 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dbba653-30e6-4e49-94a3-f666b4409c89-tmp\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.838618 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.838595 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-serving-cert\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.838747 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.838730 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-oauth-config\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.892769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.892719 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw65f\" (UniqueName: \"kubernetes.io/projected/3dbba653-30e6-4e49-94a3-f666b4409c89-kube-api-access-zw65f\") pod \"console-7ccf46c54d-zg952\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:23.956061 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.956036 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sgmhx" Nov 01 20:31:23.960573 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:23.960556 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:24.327347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:24.327167 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ccf46c54d-zg952"] Nov 01 20:31:24.329895 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:31:24.329864 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbba653_30e6_4e49_94a3_f666b4409c89.slice/crio-9cdc62e3ec43d51af0a42a4bc9a795caf40cd824dac063c6c05a756d1c56701c WatchSource:0}: Error finding container 9cdc62e3ec43d51af0a42a4bc9a795caf40cd824dac063c6c05a756d1c56701c: Status 404 returned error can't find the container with id 9cdc62e3ec43d51af0a42a4bc9a795caf40cd824dac063c6c05a756d1c56701c Nov 01 20:31:24.698907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:24.698791 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccf46c54d-zg952" event={"ID":"3dbba653-30e6-4e49-94a3-f666b4409c89","Type":"ContainerStarted","Data":"8254ba9c521654ed7ec4d9994f0d132d87a229bc73ab63dad7b85d427ccef7fd"} Nov 01 20:31:24.698907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:24.698883 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccf46c54d-zg952" event={"ID":"3dbba653-30e6-4e49-94a3-f666b4409c89","Type":"ContainerStarted","Data":"9cdc62e3ec43d51af0a42a4bc9a795caf40cd824dac063c6c05a756d1c56701c"} Nov 01 20:31:24.935011 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:24.934963 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7ccf46c54d-zg952" podStartSLOduration=1.93494452 podStartE2EDuration="1.93494452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:24.934286584 +0000 UTC m=+38.584338135" watchObservedRunningTime="2026-11-01 20:31:24.93494452 +0000 UTC m=+38.584996064" Nov 01 20:31:25.704776 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:25.704734 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerStarted","Data":"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0"} Nov 01 20:31:25.704932 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:25.704827 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerStarted","Data":"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37"} Nov 01 20:31:26.711289 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:26.711216 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerStarted","Data":"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87"} Nov 01 20:31:26.711289 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:26.711276 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerStarted","Data":"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e"} Nov 01 20:31:26.711681 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:26.711300 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerStarted","Data":"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba"} Nov 01 20:31:27.718376 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:27.718337 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerStarted","Data":"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e"} Nov 01 20:31:27.914505 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:27.914446 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.785934207 podStartE2EDuration="9.914433586s" totalImagesPullingTime="6.113619123s" totalInitContainerRuntime="14.880256ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:27.913321488 +0000 UTC m=+41.563373066" watchObservedRunningTime="2026-11-01 20:31:27.914433586 +0000 UTC m=+41.564485138" Nov 01 20:31:30.580138 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.580101 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ccf46c54d-zg952"] Nov 01 20:31:30.629596 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.629563 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ccff464f7-5vpb8"] Nov 01 20:31:30.633481 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.633446 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.673064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.673028 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ccff464f7-5vpb8"] Nov 01 20:31:30.801399 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.801365 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-oauth-config\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.801399 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.801398 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-serving-cert\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.801606 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.801425 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-service-ca\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.801606 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.801558 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-console-config\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.801606 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.801586 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-trusted-ca-bundle\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.801740 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.801606 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba13803-d541-4368-9d79-61d12b9fc11f-tmp\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.801740 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.801625 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-oauth-serving-cert\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.801740 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.801692 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzt7\" (UniqueName: \"kubernetes.io/projected/7ba13803-d541-4368-9d79-61d12b9fc11f-kube-api-access-nlzt7\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.902963 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-service-ca\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903024 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-console-config\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903044 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-trusted-ca-bundle\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903062 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba13803-d541-4368-9d79-61d12b9fc11f-tmp\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903100 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-oauth-serving-cert\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903123 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzt7\" (UniqueName: \"kubernetes.io/projected/7ba13803-d541-4368-9d79-61d12b9fc11f-kube-api-access-nlzt7\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903165 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-oauth-config\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903186 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-serving-cert\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.903677 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903653 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba13803-d541-4368-9d79-61d12b9fc11f-tmp\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.904009 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.903938 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-oauth-serving-cert\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.904009 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.904001 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-trusted-ca-bundle\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.904009 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.904016 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-console-config\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.904009 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.904116 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-service-ca\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.905895 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.905871 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-serving-cert\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.905977 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.905921 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-oauth-config\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:30.959376 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:30.959353 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzt7\" (UniqueName: \"kubernetes.io/projected/7ba13803-d541-4368-9d79-61d12b9fc11f-kube-api-access-nlzt7\") pod \"console-5ccff464f7-5vpb8\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:31.243923 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:31.243842 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:31.289131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:31.289094 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:31.612374 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:31.612220 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ccff464f7-5vpb8"] Nov 01 20:31:31.615093 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:31:31.615052 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba13803_d541_4368_9d79_61d12b9fc11f.slice/crio-39eab94ac50d56ebfc698f8c9be5895112fa5def15c3a3a9f096a98ace2a6981 WatchSource:0}: Error finding container 39eab94ac50d56ebfc698f8c9be5895112fa5def15c3a3a9f096a98ace2a6981: Status 404 returned error can't find the container with id 39eab94ac50d56ebfc698f8c9be5895112fa5def15c3a3a9f096a98ace2a6981 Nov 01 20:31:31.733730 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:31.733694 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccff464f7-5vpb8" event={"ID":"7ba13803-d541-4368-9d79-61d12b9fc11f","Type":"ContainerStarted","Data":"a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220"} Nov 01 20:31:31.733944 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:31.733754 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccff464f7-5vpb8" event={"ID":"7ba13803-d541-4368-9d79-61d12b9fc11f","Type":"ContainerStarted","Data":"39eab94ac50d56ebfc698f8c9be5895112fa5def15c3a3a9f096a98ace2a6981"} Nov 01 20:31:31.850094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:31.850020 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5ccff464f7-5vpb8" podStartSLOduration=1.85000587 podStartE2EDuration="1.85000587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:31.849872292 +0000 UTC m=+45.499923871" watchObservedRunningTime="2026-11-01 20:31:31.85000587 +0000 UTC m=+45.500057408" Nov 01 20:31:33.960725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:33.960692 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:36.552578 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:31:36.552536 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:31:41.244383 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:41.244346 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:41.244759 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:41.244401 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:41.249226 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:41.249203 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:41.772966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:41.772934 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:31:41.939774 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:41.939742 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9cfd6f878-mzshm"] Nov 01 20:31:49.721177 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:49.721106 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c8495845b-9xnpv" podUID="f5b6ce0b-61ea-4525-a727-99b1ada7f62d" containerName="console" containerID="cri-o://6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a" gracePeriod=15 Nov 01 20:31:49.974503 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:49.974446 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:50.060164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060136 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-oauth-serving-cert\") pod \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " Nov 01 20:31:50.060266 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060177 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-oauth-config\") pod \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " Nov 01 20:31:50.060266 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060197 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-config\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-config\") pod \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " Nov 01 20:31:50.060266 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060241 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-serving-cert\") pod \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " Nov 01 20:31:50.060425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060280 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-trusted-ca-bundle\") pod \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " Nov 01 20:31:50.060425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060317 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-kube-api-access-k5tn9\" (UniqueName: \"kubernetes.io/projected/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-kube-api-access-k5tn9\") pod \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " Nov 01 20:31:50.060425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060335 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-tmp\") pod \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " Nov 01 20:31:50.060425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060354 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-service-ca\") pod \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\" (UID: \"f5b6ce0b-61ea-4525-a727-99b1ada7f62d\") " Nov 01 20:31:50.060625 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060531 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-oauth-serving-cert" pod "f5b6ce0b-61ea-4525-a727-99b1ada7f62d" (UID: "f5b6ce0b-61ea-4525-a727-99b1ada7f62d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:50.060709 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060674 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-tmp" pod "f5b6ce0b-61ea-4525-a727-99b1ada7f62d" (UID: "f5b6ce0b-61ea-4525-a727-99b1ada7f62d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:50.060765 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060732 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-trusted-ca-bundle" pod "f5b6ce0b-61ea-4525-a727-99b1ada7f62d" (UID: "f5b6ce0b-61ea-4525-a727-99b1ada7f62d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:50.060838 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060820 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-config" pod "f5b6ce0b-61ea-4525-a727-99b1ada7f62d" (UID: "f5b6ce0b-61ea-4525-a727-99b1ada7f62d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:50.060961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.060944 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-service-ca" pod "f5b6ce0b-61ea-4525-a727-99b1ada7f62d" (UID: "f5b6ce0b-61ea-4525-a727-99b1ada7f62d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:50.062668 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.062645 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-kube-api-access-k5tn9" pod "f5b6ce0b-61ea-4525-a727-99b1ada7f62d" (UID: "f5b6ce0b-61ea-4525-a727-99b1ada7f62d"). InnerVolumeSpecName "kube-api-access-k5tn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:31:50.062841 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.062822 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-oauth-config" pod "f5b6ce0b-61ea-4525-a727-99b1ada7f62d" (UID: "f5b6ce0b-61ea-4525-a727-99b1ada7f62d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:31:50.062906 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.062842 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-serving-cert" pod "f5b6ce0b-61ea-4525-a727-99b1ada7f62d" (UID: "f5b6ce0b-61ea-4525-a727-99b1ada7f62d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:31:50.161133 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.161106 4477 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-trusted-ca-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.161133 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.161131 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tn9\" (UniqueName: \"kubernetes.io/projected/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-kube-api-access-k5tn9\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.161277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.161145 4477 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-tmp\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.161277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.161154 4477 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.161277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.161162 4477 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-oauth-serving-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.161277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.161172 4477 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-oauth-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.161277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.161180 4477 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.161277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.161188 4477 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5b6ce0b-61ea-4525-a727-99b1ada7f62d-console-serving-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.801516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.801426 4477 generic.go:356] "Generic (PLEG): container finished" podID="f5b6ce0b-61ea-4525-a727-99b1ada7f62d" containerID="6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a" exitCode=0 Nov 01 20:31:50.801516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.801476 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8495845b-9xnpv" Nov 01 20:31:50.801985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.801514 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8495845b-9xnpv" event={"ID":"f5b6ce0b-61ea-4525-a727-99b1ada7f62d","Type":"ContainerDied","Data":"6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a"} Nov 01 20:31:50.801985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.801563 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8495845b-9xnpv" event={"ID":"f5b6ce0b-61ea-4525-a727-99b1ada7f62d","Type":"ContainerDied","Data":"fae3d46714e8d9a9c7ed35e6728c9250a551f2aa7e2b50949aa383c3d3e52664"} Nov 01 20:31:50.801985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.801582 4477 scope.go:122] "RemoveContainer" containerID="6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a" Nov 01 20:31:50.809720 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.809704 4477 scope.go:122] "RemoveContainer" containerID="6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a" Nov 01 20:31:50.809981 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:31:50.809963 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a\": container with ID starting with 6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a not found: ID does not exist" containerID="6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a" Nov 01 20:31:50.810044 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.809993 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a"} err="failed to get container status \"6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a\": rpc error: code = NotFound desc = could not find container \"6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a\": container with ID starting with 6a1d171ceb7a0baa1dc0e8d8aa65555aac9bba365bcc89a158e228bd8a43459a not found: ID does not exist" Nov 01 20:31:50.964763 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:50.964727 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c8495845b-9xnpv"] Nov 01 20:31:51.014296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:51.014263 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c8495845b-9xnpv"] Nov 01 20:31:52.501251 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:52.501215 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b6ce0b-61ea-4525-a727-99b1ada7f62d" path="/var/lib/kubelet/pods/f5b6ce0b-61ea-4525-a727-99b1ada7f62d/volumes" Nov 01 20:31:55.601096 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.601027 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7ccf46c54d-zg952" podUID="3dbba653-30e6-4e49-94a3-f666b4409c89" containerName="console" containerID="cri-o://8254ba9c521654ed7ec4d9994f0d132d87a229bc73ab63dad7b85d427ccef7fd" gracePeriod=15 Nov 01 20:31:55.822862 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.822832 4477 generic.go:356] "Generic (PLEG): container finished" podID="3dbba653-30e6-4e49-94a3-f666b4409c89" containerID="8254ba9c521654ed7ec4d9994f0d132d87a229bc73ab63dad7b85d427ccef7fd" exitCode=0 Nov 01 20:31:55.823045 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.822872 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccf46c54d-zg952" event={"ID":"3dbba653-30e6-4e49-94a3-f666b4409c89","Type":"ContainerDied","Data":"8254ba9c521654ed7ec4d9994f0d132d87a229bc73ab63dad7b85d427ccef7fd"} Nov 01 20:31:55.861161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.861108 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:55.903335 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903305 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-oauth-config\") pod \"3dbba653-30e6-4e49-94a3-f666b4409c89\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " Nov 01 20:31:55.903468 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903361 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-trusted-ca-bundle\") pod \"3dbba653-30e6-4e49-94a3-f666b4409c89\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " Nov 01 20:31:55.903468 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903393 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-oauth-serving-cert\") pod \"3dbba653-30e6-4e49-94a3-f666b4409c89\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " Nov 01 20:31:55.903468 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903416 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-serving-cert\") pod \"3dbba653-30e6-4e49-94a3-f666b4409c89\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " Nov 01 20:31:55.903468 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903447 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbba653-30e6-4e49-94a3-f666b4409c89-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dbba653-30e6-4e49-94a3-f666b4409c89-tmp\") pod \"3dbba653-30e6-4e49-94a3-f666b4409c89\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " Nov 01 20:31:55.903657 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903478 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbba653-30e6-4e49-94a3-f666b4409c89-kube-api-access-zw65f\" (UniqueName: \"kubernetes.io/projected/3dbba653-30e6-4e49-94a3-f666b4409c89-kube-api-access-zw65f\") pod \"3dbba653-30e6-4e49-94a3-f666b4409c89\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " Nov 01 20:31:55.903657 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903517 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-service-ca\") pod \"3dbba653-30e6-4e49-94a3-f666b4409c89\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " Nov 01 20:31:55.903657 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903545 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-console-config\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-console-config\") pod \"3dbba653-30e6-4e49-94a3-f666b4409c89\" (UID: \"3dbba653-30e6-4e49-94a3-f666b4409c89\") " Nov 01 20:31:55.904055 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.903986 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbba653-30e6-4e49-94a3-f666b4409c89-tmp" pod "3dbba653-30e6-4e49-94a3-f666b4409c89" (UID: "3dbba653-30e6-4e49-94a3-f666b4409c89"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:55.904420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.904050 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-trusted-ca-bundle" pod "3dbba653-30e6-4e49-94a3-f666b4409c89" (UID: "3dbba653-30e6-4e49-94a3-f666b4409c89"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:55.904420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.904060 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-oauth-serving-cert" pod "3dbba653-30e6-4e49-94a3-f666b4409c89" (UID: "3dbba653-30e6-4e49-94a3-f666b4409c89"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:55.904534 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.904118 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-console-config" pod "3dbba653-30e6-4e49-94a3-f666b4409c89" (UID: "3dbba653-30e6-4e49-94a3-f666b4409c89"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:55.904534 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.904193 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-service-ca" pod "3dbba653-30e6-4e49-94a3-f666b4409c89" (UID: "3dbba653-30e6-4e49-94a3-f666b4409c89"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:55.905809 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.905784 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-oauth-config" pod "3dbba653-30e6-4e49-94a3-f666b4409c89" (UID: "3dbba653-30e6-4e49-94a3-f666b4409c89"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:31:55.905919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.905799 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-serving-cert" pod "3dbba653-30e6-4e49-94a3-f666b4409c89" (UID: "3dbba653-30e6-4e49-94a3-f666b4409c89"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:31:55.905919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:55.905893 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbba653-30e6-4e49-94a3-f666b4409c89-kube-api-access-zw65f" pod "3dbba653-30e6-4e49-94a3-f666b4409c89" (UID: "3dbba653-30e6-4e49-94a3-f666b4409c89"). InnerVolumeSpecName "kube-api-access-zw65f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:31:56.004610 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.004581 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw65f\" (UniqueName: \"kubernetes.io/projected/3dbba653-30e6-4e49-94a3-f666b4409c89-kube-api-access-zw65f\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:56.004610 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.004602 4477 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:56.004610 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.004611 4477 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-console-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:56.004837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.004620 4477 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-oauth-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:56.004837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.004629 4477 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-trusted-ca-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:56.004837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.004636 4477 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dbba653-30e6-4e49-94a3-f666b4409c89-oauth-serving-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:56.004837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.004645 4477 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dbba653-30e6-4e49-94a3-f666b4409c89-console-serving-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:56.004837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.004654 4477 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dbba653-30e6-4e49-94a3-f666b4409c89-tmp\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:56.827341 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.827298 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccf46c54d-zg952" event={"ID":"3dbba653-30e6-4e49-94a3-f666b4409c89","Type":"ContainerDied","Data":"9cdc62e3ec43d51af0a42a4bc9a795caf40cd824dac063c6c05a756d1c56701c"} Nov 01 20:31:56.827341 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.827341 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccf46c54d-zg952" Nov 01 20:31:56.827817 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.827366 4477 scope.go:122] "RemoveContainer" containerID="8254ba9c521654ed7ec4d9994f0d132d87a229bc73ab63dad7b85d427ccef7fd" Nov 01 20:31:56.989712 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.989683 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ccf46c54d-zg952"] Nov 01 20:31:56.994653 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:56.994637 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ccf46c54d-zg952"] Nov 01 20:31:58.501329 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:31:58.501299 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbba653-30e6-4e49-94a3-f666b4409c89" path="/var/lib/kubelet/pods/3dbba653-30e6-4e49-94a3-f666b4409c89/volumes" Nov 01 20:32:04.717090 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:04.717049 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8mzmm-bpj4x"] Nov 01 20:32:04.717542 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:04.717261 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" podUID="f6d29d9e-1400-40e2-b20b-793790117269" containerName="insights-gathering" containerID="cri-o://57ff9e150e68bd39911730ae91dc728deae0fdf0e9123d203462b1d331f431ca" gracePeriod=30 Nov 01 20:32:04.857853 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:04.857821 4477 generic.go:356] "Generic (PLEG): container finished" podID="f6d29d9e-1400-40e2-b20b-793790117269" containerID="57ff9e150e68bd39911730ae91dc728deae0fdf0e9123d203462b1d331f431ca" exitCode=2 Nov 01 20:32:04.858007 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:04.857890 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" event={"ID":"f6d29d9e-1400-40e2-b20b-793790117269","Type":"ContainerDied","Data":"57ff9e150e68bd39911730ae91dc728deae0fdf0e9123d203462b1d331f431ca"} Nov 01 20:32:04.978805 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:04.978746 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:32:05.074871 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.074838 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path\") pod \"f6d29d9e-1400-40e2-b20b-793790117269\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " Nov 01 20:32:05.075028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.074880 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs\" (UniqueName: \"kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs\") pod \"f6d29d9e-1400-40e2-b20b-793790117269\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " Nov 01 20:32:05.075028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.074901 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle\") pod \"f6d29d9e-1400-40e2-b20b-793790117269\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " Nov 01 20:32:05.075028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.074959 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle\") pod \"f6d29d9e-1400-40e2-b20b-793790117269\" (UID: \"f6d29d9e-1400-40e2-b20b-793790117269\") " Nov 01 20:32:05.075298 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.075265 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path" pod "f6d29d9e-1400-40e2-b20b-793790117269" (UID: "f6d29d9e-1400-40e2-b20b-793790117269"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:05.075421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.075315 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle" pod "f6d29d9e-1400-40e2-b20b-793790117269" (UID: "f6d29d9e-1400-40e2-b20b-793790117269"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:05.075421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.075323 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle" pod "f6d29d9e-1400-40e2-b20b-793790117269" (UID: "f6d29d9e-1400-40e2-b20b-793790117269"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:05.077249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.077229 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs" pod "f6d29d9e-1400-40e2-b20b-793790117269" (UID: "f6d29d9e-1400-40e2-b20b-793790117269"). InnerVolumeSpecName "kube-api-access-tqxvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:05.175454 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.175426 4477 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-service-ca-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.175454 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.175450 4477 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f6d29d9e-1400-40e2-b20b-793790117269-archives-path\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.175596 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.175462 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxvs\" (UniqueName: \"kubernetes.io/projected/f6d29d9e-1400-40e2-b20b-793790117269-kube-api-access-tqxvs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.175596 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.175472 4477 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6d29d9e-1400-40e2-b20b-793790117269-trusted-ca-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.862404 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.862374 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" Nov 01 20:32:05.862910 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.862369 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8mzmm-bpj4x" event={"ID":"f6d29d9e-1400-40e2-b20b-793790117269","Type":"ContainerDied","Data":"876b58cf27baf26f9cac87ab83427cb7cec697d9ba95b5dd1c2abbca0d8fd775"} Nov 01 20:32:05.862910 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:05.862501 4477 scope.go:122] "RemoveContainer" containerID="57ff9e150e68bd39911730ae91dc728deae0fdf0e9123d203462b1d331f431ca" Nov 01 20:32:06.023641 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:06.023604 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8mzmm-bpj4x"] Nov 01 20:32:06.073189 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:06.073154 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-8mzmm-bpj4x"] Nov 01 20:32:06.500822 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:06.500781 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d29d9e-1400-40e2-b20b-793790117269" path="/var/lib/kubelet/pods/f6d29d9e-1400-40e2-b20b-793790117269/volumes" Nov 01 20:32:06.552629 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:06.552590 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:32:06.960098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:06.960043 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9cfd6f878-mzshm" podUID="37cf081a-11c9-46ce-abb0-e4d20050d25f" containerName="console" containerID="cri-o://78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27" gracePeriod=15 Nov 01 20:32:07.195889 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.195868 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:32:07.289745 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.289681 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-config\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-config\") pod \"37cf081a-11c9-46ce-abb0-e4d20050d25f\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " Nov 01 20:32:07.289745 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.289720 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cf081a-11c9-46ce-abb0-e4d20050d25f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/37cf081a-11c9-46ce-abb0-e4d20050d25f-tmp\") pod \"37cf081a-11c9-46ce-abb0-e4d20050d25f\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " Nov 01 20:32:07.289919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.289751 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-oauth-config\") pod \"37cf081a-11c9-46ce-abb0-e4d20050d25f\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " Nov 01 20:32:07.289919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.289783 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cf081a-11c9-46ce-abb0-e4d20050d25f-kube-api-access-pprjr\" (UniqueName: \"kubernetes.io/projected/37cf081a-11c9-46ce-abb0-e4d20050d25f-kube-api-access-pprjr\") pod \"37cf081a-11c9-46ce-abb0-e4d20050d25f\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " Nov 01 20:32:07.289919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.289811 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-serving-cert\") pod \"37cf081a-11c9-46ce-abb0-e4d20050d25f\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " Nov 01 20:32:07.289919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.289841 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-oauth-serving-cert\") pod \"37cf081a-11c9-46ce-abb0-e4d20050d25f\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " Nov 01 20:32:07.290162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.289935 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-service-ca\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-service-ca\") pod \"37cf081a-11c9-46ce-abb0-e4d20050d25f\" (UID: \"37cf081a-11c9-46ce-abb0-e4d20050d25f\") " Nov 01 20:32:07.290162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.289994 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cf081a-11c9-46ce-abb0-e4d20050d25f-tmp" pod "37cf081a-11c9-46ce-abb0-e4d20050d25f" (UID: "37cf081a-11c9-46ce-abb0-e4d20050d25f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:07.290162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.290058 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-config" pod "37cf081a-11c9-46ce-abb0-e4d20050d25f" (UID: "37cf081a-11c9-46ce-abb0-e4d20050d25f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:07.290308 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.290294 4477 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:07.290365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.290316 4477 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37cf081a-11c9-46ce-abb0-e4d20050d25f-tmp\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:07.290365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.290331 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-oauth-serving-cert" pod "37cf081a-11c9-46ce-abb0-e4d20050d25f" (UID: "37cf081a-11c9-46ce-abb0-e4d20050d25f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:07.290365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.290349 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-service-ca" pod "37cf081a-11c9-46ce-abb0-e4d20050d25f" (UID: "37cf081a-11c9-46ce-abb0-e4d20050d25f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:07.292133 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.292110 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-serving-cert" pod "37cf081a-11c9-46ce-abb0-e4d20050d25f" (UID: "37cf081a-11c9-46ce-abb0-e4d20050d25f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:07.292235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.292130 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-oauth-config" pod "37cf081a-11c9-46ce-abb0-e4d20050d25f" (UID: "37cf081a-11c9-46ce-abb0-e4d20050d25f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:07.292235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.292157 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cf081a-11c9-46ce-abb0-e4d20050d25f-kube-api-access-pprjr" pod "37cf081a-11c9-46ce-abb0-e4d20050d25f" (UID: "37cf081a-11c9-46ce-abb0-e4d20050d25f"). InnerVolumeSpecName "kube-api-access-pprjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:07.391196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.391159 4477 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-oauth-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:07.391196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.391188 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pprjr\" (UniqueName: \"kubernetes.io/projected/37cf081a-11c9-46ce-abb0-e4d20050d25f-kube-api-access-pprjr\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:07.391196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.391199 4477 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37cf081a-11c9-46ce-abb0-e4d20050d25f-console-serving-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:07.391196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.391207 4477 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-oauth-serving-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:07.391458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.391217 4477 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37cf081a-11c9-46ce-abb0-e4d20050d25f-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:07.871555 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.871514 4477 generic.go:356] "Generic (PLEG): container finished" podID="37cf081a-11c9-46ce-abb0-e4d20050d25f" containerID="78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27" exitCode=0 Nov 01 20:32:07.871703 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.871564 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cfd6f878-mzshm" Nov 01 20:32:07.871703 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.871597 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cfd6f878-mzshm" event={"ID":"37cf081a-11c9-46ce-abb0-e4d20050d25f","Type":"ContainerDied","Data":"78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27"} Nov 01 20:32:07.871703 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.871652 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cfd6f878-mzshm" event={"ID":"37cf081a-11c9-46ce-abb0-e4d20050d25f","Type":"ContainerDied","Data":"ddb525def1c4f45e581d5c47cfd25f1961f179fc68d63ebc3dd9eccd876bfdfb"} Nov 01 20:32:07.871703 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.871672 4477 scope.go:122] "RemoveContainer" containerID="78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27" Nov 01 20:32:07.879766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.879751 4477 scope.go:122] "RemoveContainer" containerID="78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27" Nov 01 20:32:07.880040 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:07.880023 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27\": container with ID starting with 78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27 not found: ID does not exist" containerID="78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27" Nov 01 20:32:07.880114 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:07.880050 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27"} err="failed to get container status \"78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27\": rpc error: code = NotFound desc = could not find container \"78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27\": container with ID starting with 78bb1f4eb7ec9b9eeeb76472a4557e28a7fdb81aa4860b5f297f103ee9637b27 not found: ID does not exist" Nov 01 20:32:08.033830 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:08.033798 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9cfd6f878-mzshm"] Nov 01 20:32:08.083303 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:08.083273 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9cfd6f878-mzshm"] Nov 01 20:32:08.500961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:08.500933 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cf081a-11c9-46ce-abb0-e4d20050d25f" path="/var/lib/kubelet/pods/37cf081a-11c9-46ce-abb0-e4d20050d25f/volumes" Nov 01 20:32:16.266888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.266856 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c6cb84d6-ntnwr"] Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267151 4477 cpu_manager.go:414] "containerMap: removing container" podUID="37cf081a-11c9-46ce-abb0-e4d20050d25f" containerName="console" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267573 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cf081a-11c9-46ce-abb0-e4d20050d25f" containerName="console" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267591 4477 cpu_manager.go:414] "containerMap: removing container" podUID="f5b6ce0b-61ea-4525-a727-99b1ada7f62d" containerName="console" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267599 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b6ce0b-61ea-4525-a727-99b1ada7f62d" containerName="console" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267657 4477 cpu_manager.go:414] "containerMap: removing container" podUID="f6d29d9e-1400-40e2-b20b-793790117269" containerName="insights-gathering" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267665 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d29d9e-1400-40e2-b20b-793790117269" containerName="insights-gathering" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267678 4477 cpu_manager.go:414] "containerMap: removing container" podUID="3dbba653-30e6-4e49-94a3-f666b4409c89" containerName="console" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267703 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbba653-30e6-4e49-94a3-f666b4409c89" containerName="console" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267863 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d29d9e-1400-40e2-b20b-793790117269" containerName="insights-gathering" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267874 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cf081a-11c9-46ce-abb0-e4d20050d25f" containerName="console" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267891 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b6ce0b-61ea-4525-a727-99b1ada7f62d" containerName="console" Nov 01 20:32:16.269191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.267898 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbba653-30e6-4e49-94a3-f666b4409c89" containerName="console" Nov 01 20:32:16.272838 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.272815 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.349908 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.349882 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-trusted-ca-bundle\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.350022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.349918 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-config\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.350022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.349938 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-service-ca\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.350022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.349994 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6h75\" (UniqueName: \"kubernetes.io/projected/5b85e82c-b11d-4d0d-a3cf-8403cb818196-kube-api-access-n6h75\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.350164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.350114 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-oauth-serving-cert\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.350210 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.350158 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b85e82c-b11d-4d0d-a3cf-8403cb818196-tmp\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.350258 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.350246 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-serving-cert\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.350299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.350273 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-oauth-config\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.369687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.369667 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c6cb84d6-ntnwr"] Nov 01 20:32:16.451083 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451055 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-serving-cert\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451104 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-oauth-config\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451123 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-trusted-ca-bundle\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451161 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-config\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451191 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-service-ca\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451218 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6h75\" (UniqueName: \"kubernetes.io/projected/5b85e82c-b11d-4d0d-a3cf-8403cb818196-kube-api-access-n6h75\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451242 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-oauth-serving-cert\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451269 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b85e82c-b11d-4d0d-a3cf-8403cb818196-tmp\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451685 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451664 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b85e82c-b11d-4d0d-a3cf-8403cb818196-tmp\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.451990 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.451966 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-config\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.452068 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.452027 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-service-ca\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.452161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.452145 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-oauth-serving-cert\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.452299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.452280 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b85e82c-b11d-4d0d-a3cf-8403cb818196-trusted-ca-bundle\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.453611 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.453587 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-oauth-config\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.453710 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.453695 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b85e82c-b11d-4d0d-a3cf-8403cb818196-console-serving-cert\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.507593 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.507570 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6h75\" (UniqueName: \"kubernetes.io/projected/5b85e82c-b11d-4d0d-a3cf-8403cb818196-kube-api-access-n6h75\") pod \"console-78c6cb84d6-ntnwr\" (UID: \"5b85e82c-b11d-4d0d-a3cf-8403cb818196\") " pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:16.583059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:16.583032 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:18.910376 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:18.910258 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c6cb84d6-ntnwr"] Nov 01 20:32:18.914204 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:32:18.914177 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b85e82c_b11d_4d0d_a3cf_8403cb818196.slice/crio-b2aa1a9f6dfa3b0be840cf1704e0950fd6e934412164152a2de17cbb1a9c597f WatchSource:0}: Error finding container b2aa1a9f6dfa3b0be840cf1704e0950fd6e934412164152a2de17cbb1a9c597f: Status 404 returned error can't find the container with id b2aa1a9f6dfa3b0be840cf1704e0950fd6e934412164152a2de17cbb1a9c597f Nov 01 20:32:19.915552 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:19.915521 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c6cb84d6-ntnwr" event={"ID":"5b85e82c-b11d-4d0d-a3cf-8403cb818196","Type":"ContainerStarted","Data":"f12c8ef332306498d90f5307013dff7e72b95bcc2021c8691f86d5b6bee159f1"} Nov 01 20:32:19.915972 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:19.915567 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c6cb84d6-ntnwr" event={"ID":"5b85e82c-b11d-4d0d-a3cf-8403cb818196","Type":"ContainerStarted","Data":"b2aa1a9f6dfa3b0be840cf1704e0950fd6e934412164152a2de17cbb1a9c597f"} Nov 01 20:32:20.082590 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:20.082535 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78c6cb84d6-ntnwr" podStartSLOduration=4.082511218 podStartE2EDuration="4.082511218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:20.028443419 +0000 UTC m=+93.678494982" watchObservedRunningTime="2026-11-01 20:32:20.082511218 +0000 UTC m=+93.732562762" Nov 01 20:32:23.089835 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.089803 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:32:23.090369 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.090098 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="alertmanager" containerID="cri-o://d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37" gracePeriod=120 Nov 01 20:32:23.090369 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.090171 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy-metric" containerID="cri-o://bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87" gracePeriod=120 Nov 01 20:32:23.090369 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.090201 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="config-reloader" containerID="cri-o://f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0" gracePeriod=120 Nov 01 20:32:23.090369 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.090238 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy" containerID="cri-o://8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e" gracePeriod=120 Nov 01 20:32:23.090369 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.090230 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="prom-label-proxy" containerID="cri-o://c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e" gracePeriod=120 Nov 01 20:32:23.090369 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.090182 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy-web" containerID="cri-o://80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba" gracePeriod=120 Nov 01 20:32:23.933921 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.933887 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerID="c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e" exitCode=0 Nov 01 20:32:23.933921 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.933910 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerID="8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e" exitCode=0 Nov 01 20:32:23.933921 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.933916 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerID="f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0" exitCode=0 Nov 01 20:32:23.933921 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.933922 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerID="d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37" exitCode=0 Nov 01 20:32:23.934214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.933956 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerDied","Data":"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e"} Nov 01 20:32:23.934214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.934017 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerDied","Data":"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e"} Nov 01 20:32:23.934214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.934043 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerDied","Data":"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0"} Nov 01 20:32:23.934214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:23.934066 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerDied","Data":"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37"} Nov 01 20:32:24.332206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.332183 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:24.412241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412208 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.412241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412242 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-main-tls\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.412437 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412284 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-config-volume\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-config-volume\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.412437 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412391 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-cluster-tls-config\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.412437 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412429 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-web-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-web-config\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.412565 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412482 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-web\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.412565 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412517 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-trusted-ca-bundle\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.412565 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412551 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-config-out\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.413352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412590 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-main-db\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.413352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412621 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-metrics-client-ca\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.413352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412669 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-tls-assets\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-tls-assets\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.413352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412703 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.413352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.412738 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-kube-api-access-t9k2l\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-kube-api-access-t9k2l\") pod \"a9199082-eca3-44a0-8a29-5398460a1ca7\" (UID: \"a9199082-eca3-44a0-8a29-5398460a1ca7\") " Nov 01 20:32:24.413352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.413252 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-main-db" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:24.413662 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.413604 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-trusted-ca-bundle" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:24.414127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.414102 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-metrics-client-ca" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:24.415353 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.415313 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-config-volume" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:24.416214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.416185 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-kube-api-access-t9k2l" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "kube-api-access-t9k2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:24.416214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.416193 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-metric" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:24.416214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.416204 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:24.416437 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.416231 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-main-tls" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:24.416499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.416481 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-web" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:24.417375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.417353 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-tls-assets" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:24.417609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.417595 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-config-out" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:24.420326 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.420258 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-cluster-tls-config" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:24.426555 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.426535 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-web-config" pod "a9199082-eca3-44a0-8a29-5398460a1ca7" (UID: "a9199082-eca3-44a0-8a29-5398460a1ca7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:24.513379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513361 4477 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513381 4477 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-main-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513392 4477 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-config-volume\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513401 4477 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-cluster-tls-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513410 4477 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-web-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513418 4477 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513426 4477 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513434 4477 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-config-out\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513442 4477 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9199082-eca3-44a0-8a29-5398460a1ca7-alertmanager-main-db\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513451 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9199082-eca3-44a0-8a29-5398460a1ca7-metrics-client-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513458 4477 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-tls-assets\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513736 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513467 4477 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9199082-eca3-44a0-8a29-5398460a1ca7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.513736 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.513476 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9k2l\" (UniqueName: \"kubernetes.io/projected/a9199082-eca3-44a0-8a29-5398460a1ca7-kube-api-access-t9k2l\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:24.940679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.940647 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerID="bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87" exitCode=0 Nov 01 20:32:24.940679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.940674 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerID="80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba" exitCode=0 Nov 01 20:32:24.940890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.940701 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerDied","Data":"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87"} Nov 01 20:32:24.940890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.940731 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:24.940890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.940784 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerDied","Data":"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba"} Nov 01 20:32:24.940890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.940812 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9199082-eca3-44a0-8a29-5398460a1ca7","Type":"ContainerDied","Data":"bb3935bb27d7f6fede1e7eff2d102ae124156878df2627173ed5e15e8a33e87d"} Nov 01 20:32:24.940890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.940840 4477 scope.go:122] "RemoveContainer" containerID="c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e" Nov 01 20:32:24.948290 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.948269 4477 scope.go:122] "RemoveContainer" containerID="bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87" Nov 01 20:32:24.955479 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.955457 4477 scope.go:122] "RemoveContainer" containerID="8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e" Nov 01 20:32:24.961991 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.961974 4477 scope.go:122] "RemoveContainer" containerID="80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba" Nov 01 20:32:24.968262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.968249 4477 scope.go:122] "RemoveContainer" containerID="f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0" Nov 01 20:32:24.974580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.974565 4477 scope.go:122] "RemoveContainer" containerID="d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37" Nov 01 20:32:24.980944 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.980922 4477 scope.go:122] "RemoveContainer" containerID="efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417" Nov 01 20:32:24.987766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.987745 4477 scope.go:122] "RemoveContainer" containerID="c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e" Nov 01 20:32:24.988044 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:24.988025 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e\": container with ID starting with c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e not found: ID does not exist" containerID="c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e" Nov 01 20:32:24.988107 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.988057 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e"} err="failed to get container status \"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e\": rpc error: code = NotFound desc = could not find container \"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e\": container with ID starting with c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e not found: ID does not exist" Nov 01 20:32:24.988107 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.988095 4477 scope.go:122] "RemoveContainer" containerID="bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87" Nov 01 20:32:24.988372 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:24.988354 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87\": container with ID starting with bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87 not found: ID does not exist" containerID="bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87" Nov 01 20:32:24.988425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.988378 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87"} err="failed to get container status \"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87\": rpc error: code = NotFound desc = could not find container \"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87\": container with ID starting with bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87 not found: ID does not exist" Nov 01 20:32:24.988425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.988395 4477 scope.go:122] "RemoveContainer" containerID="8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e" Nov 01 20:32:24.988623 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:24.988608 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e\": container with ID starting with 8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e not found: ID does not exist" containerID="8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e" Nov 01 20:32:24.988665 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.988628 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e"} err="failed to get container status \"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e\": rpc error: code = NotFound desc = could not find container \"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e\": container with ID starting with 8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e not found: ID does not exist" Nov 01 20:32:24.988665 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.988640 4477 scope.go:122] "RemoveContainer" containerID="80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba" Nov 01 20:32:24.988879 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:24.988862 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba\": container with ID starting with 80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba not found: ID does not exist" containerID="80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba" Nov 01 20:32:24.988914 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.988885 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba"} err="failed to get container status \"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba\": rpc error: code = NotFound desc = could not find container \"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba\": container with ID starting with 80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba not found: ID does not exist" Nov 01 20:32:24.988914 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.988899 4477 scope.go:122] "RemoveContainer" containerID="f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0" Nov 01 20:32:24.989126 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:24.989112 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0\": container with ID starting with f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0 not found: ID does not exist" containerID="f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0" Nov 01 20:32:24.989172 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989130 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0"} err="failed to get container status \"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0\": rpc error: code = NotFound desc = could not find container \"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0\": container with ID starting with f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0 not found: ID does not exist" Nov 01 20:32:24.989172 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989145 4477 scope.go:122] "RemoveContainer" containerID="d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37" Nov 01 20:32:24.989352 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:24.989336 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37\": container with ID starting with d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37 not found: ID does not exist" containerID="d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37" Nov 01 20:32:24.989390 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989359 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37"} err="failed to get container status \"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37\": rpc error: code = NotFound desc = could not find container \"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37\": container with ID starting with d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37 not found: ID does not exist" Nov 01 20:32:24.989390 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989372 4477 scope.go:122] "RemoveContainer" containerID="efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417" Nov 01 20:32:24.989570 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:24.989556 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417\": container with ID starting with efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417 not found: ID does not exist" containerID="efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417" Nov 01 20:32:24.989609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989573 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417"} err="failed to get container status \"efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417\": rpc error: code = NotFound desc = could not find container \"efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417\": container with ID starting with efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417 not found: ID does not exist" Nov 01 20:32:24.989609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989583 4477 scope.go:122] "RemoveContainer" containerID="c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e" Nov 01 20:32:24.989784 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989766 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e"} err="failed to get container status \"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e\": rpc error: code = NotFound desc = could not find container \"c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e\": container with ID starting with c8073358067256e8d5881cbf923e8864e2a2cd1d7f2969a40380825b2264973e not found: ID does not exist" Nov 01 20:32:24.989826 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989784 4477 scope.go:122] "RemoveContainer" containerID="bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87" Nov 01 20:32:24.989996 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989982 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87"} err="failed to get container status \"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87\": rpc error: code = NotFound desc = could not find container \"bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87\": container with ID starting with bfff463eb4a739bb128d10643de4ed077894f7c0cfbf6104e3c315c2b9ea1c87 not found: ID does not exist" Nov 01 20:32:24.990036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.989996 4477 scope.go:122] "RemoveContainer" containerID="8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e" Nov 01 20:32:24.990255 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.990228 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e"} err="failed to get container status \"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e\": rpc error: code = NotFound desc = could not find container \"8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e\": container with ID starting with 8042795a1f7d00bd2f0581d74a3df24949f1f665129ca94f5b643d8fa64ac91e not found: ID does not exist" Nov 01 20:32:24.990255 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.990250 4477 scope.go:122] "RemoveContainer" containerID="80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba" Nov 01 20:32:24.990492 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.990474 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba"} err="failed to get container status \"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba\": rpc error: code = NotFound desc = could not find container \"80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba\": container with ID starting with 80b5414bb28de966f5fc0c5c4889ef9cff8a6efad7a4fb930b3372ec0b0650ba not found: ID does not exist" Nov 01 20:32:24.990574 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.990493 4477 scope.go:122] "RemoveContainer" containerID="f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0" Nov 01 20:32:24.990733 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.990714 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0"} err="failed to get container status \"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0\": rpc error: code = NotFound desc = could not find container \"f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0\": container with ID starting with f8210defd5c5d57c31ec1bc324a3889af52e533a9f7e8698d92aa7f118cc81d0 not found: ID does not exist" Nov 01 20:32:24.990778 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.990733 4477 scope.go:122] "RemoveContainer" containerID="d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37" Nov 01 20:32:24.990933 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.990915 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37"} err="failed to get container status \"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37\": rpc error: code = NotFound desc = could not find container \"d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37\": container with ID starting with d17895eef307053c142f5e6ba2e65b80611052be75a8e4bb5d28256fac491c37 not found: ID does not exist" Nov 01 20:32:24.990976 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.990943 4477 scope.go:122] "RemoveContainer" containerID="efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417" Nov 01 20:32:24.991150 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:24.991128 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417"} err="failed to get container status \"efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417\": rpc error: code = NotFound desc = could not find container \"efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417\": container with ID starting with efc4f3e41cf193fd8dfdba805409e02ece168670fe65c42f469a0fe34a645417 not found: ID does not exist" Nov 01 20:32:25.105640 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.105598 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:32:25.156342 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.156296 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:32:25.156755 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.156736 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:32:25.156979 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.156967 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="prom-label-proxy" Nov 01 20:32:25.157017 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.156980 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="prom-label-proxy" Nov 01 20:32:25.157017 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.156989 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="init-config-reloader" Nov 01 20:32:25.157017 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.156994 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="init-config-reloader" Nov 01 20:32:25.157017 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157002 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="alertmanager" Nov 01 20:32:25.157017 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157007 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="alertmanager" Nov 01 20:32:25.157017 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157013 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="config-reloader" Nov 01 20:32:25.157017 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157017 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="config-reloader" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157025 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy-metric" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157030 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy-metric" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157038 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy-web" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157042 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy-web" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157059 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157063 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157117 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157124 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="prom-label-proxy" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157132 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="alertmanager" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157137 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="config-reloader" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157142 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy-metric" Nov 01 20:32:25.157260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.157148 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" containerName="kube-rbac-proxy-web" Nov 01 20:32:25.162162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.162144 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.217902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217779 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 20:32:25.217902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217779 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 20:32:25.217902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217786 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 20:32:25.217902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217788 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 20:32:25.217902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217821 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w8s42\"" Nov 01 20:32:25.217902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217893 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 20:32:25.218210 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217930 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 20:32:25.218210 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217931 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 20:32:25.218210 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.217942 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 20:32:25.219659 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219643 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9d8597c-4293-4179-a45e-1d221615a5b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.219709 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219672 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.219709 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219693 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d8597c-4293-4179-a45e-1d221615a5b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.219778 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219715 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.219827 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219810 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.219862 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219840 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-web-config\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.219897 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219859 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvzr\" (UniqueName: \"kubernetes.io/projected/c9d8597c-4293-4179-a45e-1d221615a5b3-kube-api-access-zbvzr\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.219953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219940 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.220012 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.219996 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9d8597c-4293-4179-a45e-1d221615a5b3-config-out\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.220046 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.220023 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9d8597c-4293-4179-a45e-1d221615a5b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.220046 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.220043 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.220147 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.220066 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.220147 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.220112 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9d8597c-4293-4179-a45e-1d221615a5b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.267624 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.267593 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 20:32:25.271827 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.271794 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:32:25.321440 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321407 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321440 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321439 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9d8597c-4293-4179-a45e-1d221615a5b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321457 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9d8597c-4293-4179-a45e-1d221615a5b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321480 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321545 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d8597c-4293-4179-a45e-1d221615a5b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321595 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321624 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321917 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321649 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-web-config\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321917 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321676 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvzr\" (UniqueName: \"kubernetes.io/projected/c9d8597c-4293-4179-a45e-1d221615a5b3-kube-api-access-zbvzr\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321917 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321741 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321917 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321777 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9d8597c-4293-4179-a45e-1d221615a5b3-config-out\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321917 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321805 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9d8597c-4293-4179-a45e-1d221615a5b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.321917 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.321835 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.322516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.322309 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9d8597c-4293-4179-a45e-1d221615a5b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.322640 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.322616 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d8597c-4293-4179-a45e-1d221615a5b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.323272 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.323247 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9d8597c-4293-4179-a45e-1d221615a5b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.324953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.324927 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9d8597c-4293-4179-a45e-1d221615a5b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.325048 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.324953 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.325319 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.325297 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.325375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.325308 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-web-config\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.325414 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.325390 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.325582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.325561 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.326135 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.326121 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.326305 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.326291 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9d8597c-4293-4179-a45e-1d221615a5b3-config-out\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.326764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.326748 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9d8597c-4293-4179-a45e-1d221615a5b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.378654 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.378617 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvzr\" (UniqueName: \"kubernetes.io/projected/c9d8597c-4293-4179-a45e-1d221615a5b3-kube-api-access-zbvzr\") pod \"alertmanager-main-0\" (UID: \"c9d8597c-4293-4179-a45e-1d221615a5b3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.486170 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.486066 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:32:25.963829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:25.963802 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:32:25.966145 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:32:25.966118 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d8597c_4293_4179_a45e_1d221615a5b3.slice/crio-d069377cc41fd1ddbb12cdceaa6d2f12af35bdb45f2e3a0965e0bc53bc3d26eb WatchSource:0}: Error finding container d069377cc41fd1ddbb12cdceaa6d2f12af35bdb45f2e3a0965e0bc53bc3d26eb: Status 404 returned error can't find the container with id d069377cc41fd1ddbb12cdceaa6d2f12af35bdb45f2e3a0965e0bc53bc3d26eb Nov 01 20:32:26.501769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:26.501734 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9199082-eca3-44a0-8a29-5398460a1ca7" path="/var/lib/kubelet/pods/a9199082-eca3-44a0-8a29-5398460a1ca7/volumes" Nov 01 20:32:26.583243 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:26.583209 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:26.583420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:26.583274 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:26.588496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:26.588470 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:26.951417 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:26.951374 4477 generic.go:356] "Generic (PLEG): container finished" podID="c9d8597c-4293-4179-a45e-1d221615a5b3" containerID="5edc1e0ae9043154edd53f1eb5a5f6a15dcbc23677c4bbd37a19f0f78be4b643" exitCode=0 Nov 01 20:32:26.951629 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:26.951468 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d8597c-4293-4179-a45e-1d221615a5b3","Type":"ContainerDied","Data":"5edc1e0ae9043154edd53f1eb5a5f6a15dcbc23677c4bbd37a19f0f78be4b643"} Nov 01 20:32:26.951629 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:26.951543 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d8597c-4293-4179-a45e-1d221615a5b3","Type":"ContainerStarted","Data":"d069377cc41fd1ddbb12cdceaa6d2f12af35bdb45f2e3a0965e0bc53bc3d26eb"} Nov 01 20:32:26.955593 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:26.955571 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78c6cb84d6-ntnwr" Nov 01 20:32:27.212444 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:27.212417 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ccff464f7-5vpb8"] Nov 01 20:32:27.962851 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:27.962815 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d8597c-4293-4179-a45e-1d221615a5b3","Type":"ContainerStarted","Data":"c7558bf3b75edcb54b1909a567136fd8de69e90533e3f13a589e1006746cdb57"} Nov 01 20:32:27.963355 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:27.962929 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d8597c-4293-4179-a45e-1d221615a5b3","Type":"ContainerStarted","Data":"903185e1e8a01109cc828476079fdeead8198f69a41b5fe930e0193d64a0c64b"} Nov 01 20:32:27.963355 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:27.962973 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d8597c-4293-4179-a45e-1d221615a5b3","Type":"ContainerStarted","Data":"55a107482196977ea165947d1e0ac82110162f87b18f2d8343bb3676ae9970e1"} Nov 01 20:32:27.963355 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:27.963021 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d8597c-4293-4179-a45e-1d221615a5b3","Type":"ContainerStarted","Data":"dc401d64da4c0cc855d4d9f1a46d2cc0c826ec7d3d788c25079141c8deb80f0a"} Nov 01 20:32:27.963355 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:27.963058 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d8597c-4293-4179-a45e-1d221615a5b3","Type":"ContainerStarted","Data":"9e84bc7f34e0f6002d13349bbc08f8e96f4f465f773e101cc999ab1160cd7b59"} Nov 01 20:32:27.963355 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:27.963115 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d8597c-4293-4179-a45e-1d221615a5b3","Type":"ContainerStarted","Data":"3916dfd6b899e642d0a7a501fb0d8fe3e3580d6dacc0a49acedc31cbbc2c43d3"} Nov 01 20:32:28.084372 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:28.084322 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.071983624 podStartE2EDuration="3.084305622s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.321998ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:28.083279709 +0000 UTC m=+101.733331266" watchObservedRunningTime="2026-11-01 20:32:28.084305622 +0000 UTC m=+101.734357167" Nov 01 20:32:36.551054 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:36.551019 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:32:52.234450 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.234392 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5ccff464f7-5vpb8" podUID="7ba13803-d541-4368-9d79-61d12b9fc11f" containerName="console" containerID="cri-o://a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220" gracePeriod=15 Nov 01 20:32:52.486153 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.486093 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:32:52.645194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645163 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-serving-cert\") pod \"7ba13803-d541-4368-9d79-61d12b9fc11f\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " Nov 01 20:32:52.645356 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645215 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-oauth-serving-cert\") pod \"7ba13803-d541-4368-9d79-61d12b9fc11f\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " Nov 01 20:32:52.645356 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645248 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba13803-d541-4368-9d79-61d12b9fc11f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba13803-d541-4368-9d79-61d12b9fc11f-tmp\") pod \"7ba13803-d541-4368-9d79-61d12b9fc11f\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " Nov 01 20:32:52.645479 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645406 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-trusted-ca-bundle\") pod \"7ba13803-d541-4368-9d79-61d12b9fc11f\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " Nov 01 20:32:52.645479 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645461 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba13803-d541-4368-9d79-61d12b9fc11f-kube-api-access-nlzt7\" (UniqueName: \"kubernetes.io/projected/7ba13803-d541-4368-9d79-61d12b9fc11f-kube-api-access-nlzt7\") pod \"7ba13803-d541-4368-9d79-61d12b9fc11f\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " Nov 01 20:32:52.645583 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645490 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba13803-d541-4368-9d79-61d12b9fc11f-tmp" pod "7ba13803-d541-4368-9d79-61d12b9fc11f" (UID: "7ba13803-d541-4368-9d79-61d12b9fc11f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:52.645583 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645502 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-service-ca\") pod \"7ba13803-d541-4368-9d79-61d12b9fc11f\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " Nov 01 20:32:52.645583 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645527 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-console-config\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-console-config\") pod \"7ba13803-d541-4368-9d79-61d12b9fc11f\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " Nov 01 20:32:52.645583 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645567 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-oauth-config\") pod \"7ba13803-d541-4368-9d79-61d12b9fc11f\" (UID: \"7ba13803-d541-4368-9d79-61d12b9fc11f\") " Nov 01 20:32:52.645779 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645745 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-trusted-ca-bundle" pod "7ba13803-d541-4368-9d79-61d12b9fc11f" (UID: "7ba13803-d541-4368-9d79-61d12b9fc11f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:52.645779 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645762 4477 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba13803-d541-4368-9d79-61d12b9fc11f-tmp\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:52.645779 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645759 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-oauth-serving-cert" pod "7ba13803-d541-4368-9d79-61d12b9fc11f" (UID: "7ba13803-d541-4368-9d79-61d12b9fc11f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:52.645907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645796 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-service-ca" pod "7ba13803-d541-4368-9d79-61d12b9fc11f" (UID: "7ba13803-d541-4368-9d79-61d12b9fc11f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:52.645995 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.645973 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-console-config" pod "7ba13803-d541-4368-9d79-61d12b9fc11f" (UID: "7ba13803-d541-4368-9d79-61d12b9fc11f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:52.647577 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.647552 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-serving-cert" pod "7ba13803-d541-4368-9d79-61d12b9fc11f" (UID: "7ba13803-d541-4368-9d79-61d12b9fc11f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:52.647852 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.647825 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba13803-d541-4368-9d79-61d12b9fc11f-kube-api-access-nlzt7" pod "7ba13803-d541-4368-9d79-61d12b9fc11f" (UID: "7ba13803-d541-4368-9d79-61d12b9fc11f"). InnerVolumeSpecName "kube-api-access-nlzt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:52.647852 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.647839 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-oauth-config" pod "7ba13803-d541-4368-9d79-61d12b9fc11f" (UID: "7ba13803-d541-4368-9d79-61d12b9fc11f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:52.746886 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.746824 4477 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-trusted-ca-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:52.746886 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.746856 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzt7\" (UniqueName: \"kubernetes.io/projected/7ba13803-d541-4368-9d79-61d12b9fc11f-kube-api-access-nlzt7\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:52.746886 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.746867 4477 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:52.746886 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.746875 4477 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-console-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:52.746886 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.746884 4477 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-oauth-config\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:52.746886 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.746893 4477 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ba13803-d541-4368-9d79-61d12b9fc11f-console-serving-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:52.747174 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:52.746902 4477 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ba13803-d541-4368-9d79-61d12b9fc11f-oauth-serving-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:53.049503 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.049412 4477 generic.go:356] "Generic (PLEG): container finished" podID="7ba13803-d541-4368-9d79-61d12b9fc11f" containerID="a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220" exitCode=0 Nov 01 20:32:53.049503 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.049461 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccff464f7-5vpb8" event={"ID":"7ba13803-d541-4368-9d79-61d12b9fc11f","Type":"ContainerDied","Data":"a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220"} Nov 01 20:32:53.049739 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.049510 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccff464f7-5vpb8" event={"ID":"7ba13803-d541-4368-9d79-61d12b9fc11f","Type":"ContainerDied","Data":"39eab94ac50d56ebfc698f8c9be5895112fa5def15c3a3a9f096a98ace2a6981"} Nov 01 20:32:53.049739 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.049465 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccff464f7-5vpb8" Nov 01 20:32:53.049739 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.049537 4477 scope.go:122] "RemoveContainer" containerID="a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220" Nov 01 20:32:53.058231 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.058211 4477 scope.go:122] "RemoveContainer" containerID="a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220" Nov 01 20:32:53.058506 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:32:53.058485 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220\": container with ID starting with a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220 not found: ID does not exist" containerID="a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220" Nov 01 20:32:53.058566 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.058516 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220"} err="failed to get container status \"a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220\": rpc error: code = NotFound desc = could not find container \"a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220\": container with ID starting with a0b16da79d7330890629349c845f37180d60f88e7dd03a6d645914f8b360b220 not found: ID does not exist" Nov 01 20:32:53.212270 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.212236 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ccff464f7-5vpb8"] Nov 01 20:32:53.261536 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:53.261494 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ccff464f7-5vpb8"] Nov 01 20:32:54.500599 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:32:54.500563 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba13803-d541-4368-9d79-61d12b9fc11f" path="/var/lib/kubelet/pods/7ba13803-d541-4368-9d79-61d12b9fc11f/volumes" Nov 01 20:33:06.552057 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:33:06.551973 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:33:36.551689 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:33:36.551653 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:34:06.552302 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:34:06.552260 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:34:08.985759 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:08.985720 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk"] Nov 01 20:34:08.986253 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:08.986091 4477 cpu_manager.go:414] "containerMap: removing container" podUID="7ba13803-d541-4368-9d79-61d12b9fc11f" containerName="console" Nov 01 20:34:08.986253 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:08.986112 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba13803-d541-4368-9d79-61d12b9fc11f" containerName="console" Nov 01 20:34:08.986253 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:08.986188 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba13803-d541-4368-9d79-61d12b9fc11f" containerName="console" Nov 01 20:34:08.989305 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:08.989284 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.096338 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.096299 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-6cnww\"" Nov 01 20:34:09.096542 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.096355 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 20:34:09.096542 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.096366 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 20:34:09.097346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.097321 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk"] Nov 01 20:34:09.104827 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.104806 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.104940 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.104844 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/8b16c78d-103c-438e-ad5d-103fc6e68ac2-kube-api-access-gjrb7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.104979 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.104935 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.109306 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.109283 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s"] Nov 01 20:34:09.112650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.112634 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.164447 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.164411 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-w6zs7\"" Nov 01 20:34:09.205802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.205765 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/8b16c78d-103c-438e-ad5d-103fc6e68ac2-kube-api-access-gjrb7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.205970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.205831 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.205970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.205865 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.206244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.206228 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.206288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.206268 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.258244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.258161 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s"] Nov 01 20:34:09.266062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.266032 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/8b16c78d-103c-438e-ad5d-103fc6e68ac2-kube-api-access-gjrb7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.304711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.304686 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:09.307059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.307037 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfkz\" (UniqueName: \"kubernetes.io/projected/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-kube-api-access-vnfkz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.307143 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.307096 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.307267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.307246 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.408090 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.408049 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.408267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.408126 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfkz\" (UniqueName: \"kubernetes.io/projected/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-kube-api-access-vnfkz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.408267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.408167 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.408443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.408422 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.408481 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.408442 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.464674 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.464645 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfkz\" (UniqueName: \"kubernetes.io/projected/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-kube-api-access-vnfkz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:09.677784 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.677747 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk"] Nov 01 20:34:09.681338 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:09.681309 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b16c78d_103c_438e_ad5d_103fc6e68ac2.slice/crio-fa13da994877af4540d0821f4db60aabc9f716d97c9d3fbcd484545144e6dd64 WatchSource:0}: Error finding container fa13da994877af4540d0821f4db60aabc9f716d97c9d3fbcd484545144e6dd64: Status 404 returned error can't find the container with id fa13da994877af4540d0821f4db60aabc9f716d97c9d3fbcd484545144e6dd64 Nov 01 20:34:09.728218 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:09.728186 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:10.098983 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:10.098946 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s"] Nov 01 20:34:10.102508 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:10.102476 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a978b7c_ca9f_4391_b26d_53ba29f18c2e.slice/crio-d269b9e6bd3fd0b43f2c3a73fee5ef6c0a631c80eed11fdf480d87fb2b4288cf WatchSource:0}: Error finding container d269b9e6bd3fd0b43f2c3a73fee5ef6c0a631c80eed11fdf480d87fb2b4288cf: Status 404 returned error can't find the container with id d269b9e6bd3fd0b43f2c3a73fee5ef6c0a631c80eed11fdf480d87fb2b4288cf Nov 01 20:34:10.302320 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:10.302286 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" event={"ID":"3a978b7c-ca9f-4391-b26d-53ba29f18c2e","Type":"ContainerStarted","Data":"d269b9e6bd3fd0b43f2c3a73fee5ef6c0a631c80eed11fdf480d87fb2b4288cf"} Nov 01 20:34:10.303533 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:10.303506 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" event={"ID":"8b16c78d-103c-438e-ad5d-103fc6e68ac2","Type":"ContainerStarted","Data":"fa13da994877af4540d0821f4db60aabc9f716d97c9d3fbcd484545144e6dd64"} Nov 01 20:34:11.942180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:11.942144 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c"] Nov 01 20:34:11.945462 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:11.945442 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.030572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.030542 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.030741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.030590 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.030741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.030613 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbnf9\" (UniqueName: \"kubernetes.io/projected/12b6dd46-9985-44f9-a905-d904087774d1-kube-api-access-dbnf9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.046994 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.046963 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ggwrf\"" Nov 01 20:34:12.048286 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.048210 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c"] Nov 01 20:34:12.131123 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.131085 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.131318 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.131133 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbnf9\" (UniqueName: \"kubernetes.io/projected/12b6dd46-9985-44f9-a905-d904087774d1-kube-api-access-dbnf9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.131318 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.131206 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.131534 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.131511 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.131534 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.131530 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.189588 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.189552 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbnf9\" (UniqueName: \"kubernetes.io/projected/12b6dd46-9985-44f9-a905-d904087774d1-kube-api-access-dbnf9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.262195 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.262101 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:12.644975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.644942 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c"] Nov 01 20:34:12.648448 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:12.648416 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b6dd46_9985_44f9_a905_d904087774d1.slice/crio-7251af55f1a848ca54245f11ab05d80bc1fc52c482021869f4eb233e80c20be5 WatchSource:0}: Error finding container 7251af55f1a848ca54245f11ab05d80bc1fc52c482021869f4eb233e80c20be5: Status 404 returned error can't find the container with id 7251af55f1a848ca54245f11ab05d80bc1fc52c482021869f4eb233e80c20be5 Nov 01 20:34:12.961967 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.961890 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc"] Nov 01 20:34:12.967047 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:12.967023 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.021895 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.021865 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-kt2mk\"" Nov 01 20:34:13.022832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.022810 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc"] Nov 01 20:34:13.037343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.037314 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9p6\" (UniqueName: \"kubernetes.io/projected/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-kube-api-access-nt9p6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.037491 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.037356 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.037491 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.037462 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.059279 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.059250 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67"] Nov 01 20:34:13.062829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.062810 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.119508 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.119475 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67"] Nov 01 20:34:13.120902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.120885 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-g8q82\"" Nov 01 20:34:13.137901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.137874 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.138096 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.137914 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9p6\" (UniqueName: \"kubernetes.io/projected/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-kube-api-access-nt9p6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.138096 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.137935 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.138096 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.137965 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.138096 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.137989 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.138292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.138126 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc22t\" (UniqueName: \"kubernetes.io/projected/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-kube-api-access-sc22t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.138347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.138331 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.138382 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.138351 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.158688 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.158655 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8"] Nov 01 20:34:13.162412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.162389 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.197395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.197364 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9p6\" (UniqueName: \"kubernetes.io/projected/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-kube-api-access-nt9p6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.219357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.219274 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8"] Nov 01 20:34:13.220907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.220878 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5rvtz\"" Nov 01 20:34:13.238699 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.238666 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.238699 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.238701 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.238953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.238752 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.238953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.238791 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc22t\" (UniqueName: \"kubernetes.io/projected/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-kube-api-access-sc22t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.238953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.238816 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xk4d\" (UniqueName: \"kubernetes.io/projected/84010e72-32ff-487e-ad03-7c6608757314-kube-api-access-8xk4d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.238953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.238844 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.239183 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.239164 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.239240 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.239208 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.269797 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.269765 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5"] Nov 01 20:34:13.273199 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.273184 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.286619 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.286594 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:13.295930 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.295908 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc22t\" (UniqueName: \"kubernetes.io/projected/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-kube-api-access-sc22t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.320977 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.320946 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" event={"ID":"12b6dd46-9985-44f9-a905-d904087774d1","Type":"ContainerStarted","Data":"7251af55f1a848ca54245f11ab05d80bc1fc52c482021869f4eb233e80c20be5"} Nov 01 20:34:13.327328 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.327297 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-zw492\"" Nov 01 20:34:13.329855 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.329829 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5"] Nov 01 20:34:13.339727 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.339695 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xk4d\" (UniqueName: \"kubernetes.io/projected/84010e72-32ff-487e-ad03-7c6608757314-kube-api-access-8xk4d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.339896 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.339741 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.339896 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.339767 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.339896 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.339811 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.339896 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.339886 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.340118 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.339957 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7tt\" (UniqueName: \"kubernetes.io/projected/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-kube-api-access-mj7tt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.340189 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.340174 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.340224 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.340209 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.382394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.382367 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:13.397953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.397929 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xk4d\" (UniqueName: \"kubernetes.io/projected/84010e72-32ff-487e-ad03-7c6608757314-kube-api-access-8xk4d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.440816 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.440788 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.441143 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.440823 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7tt\" (UniqueName: \"kubernetes.io/projected/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-kube-api-access-mj7tt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.441143 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.440861 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.441270 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.441213 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.441349 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.441326 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.481812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.481740 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:13.499455 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.499428 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7tt\" (UniqueName: \"kubernetes.io/projected/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-kube-api-access-mj7tt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.590713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.590681 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:13.672060 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.670619 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc"] Nov 01 20:34:13.681465 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:13.681434 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967c85c5_ef28_4ebc_bb49_62f0df3c23e9.slice/crio-97ecbb0446f29ae374a5c49a4f770ad4c4bff0aae0ce90b7e35e9b0e71c8b1ea WatchSource:0}: Error finding container 97ecbb0446f29ae374a5c49a4f770ad4c4bff0aae0ce90b7e35e9b0e71c8b1ea: Status 404 returned error can't find the container with id 97ecbb0446f29ae374a5c49a4f770ad4c4bff0aae0ce90b7e35e9b0e71c8b1ea Nov 01 20:34:13.769138 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.760921 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67"] Nov 01 20:34:13.773142 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:13.773110 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8d31f0_4616_4e2f_b4dc_4a0af73bf1f3.slice/crio-1843f718fb8f6d1ec203106d1c3b7f9a008b766abb8798cdcf5f48e0268bb2f1 WatchSource:0}: Error finding container 1843f718fb8f6d1ec203106d1c3b7f9a008b766abb8798cdcf5f48e0268bb2f1: Status 404 returned error can't find the container with id 1843f718fb8f6d1ec203106d1c3b7f9a008b766abb8798cdcf5f48e0268bb2f1 Nov 01 20:34:13.862231 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.862201 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8"] Nov 01 20:34:13.866805 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:13.866772 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84010e72_32ff_487e_ad03_7c6608757314.slice/crio-afc3604c862fc0a99c36f7b6c30139c3fab7f223245b394a3c49905a012f732c WatchSource:0}: Error finding container afc3604c862fc0a99c36f7b6c30139c3fab7f223245b394a3c49905a012f732c: Status 404 returned error can't find the container with id afc3604c862fc0a99c36f7b6c30139c3fab7f223245b394a3c49905a012f732c Nov 01 20:34:13.976257 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:13.976224 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5"] Nov 01 20:34:13.980629 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:13.980593 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce8176c_9f10_41d0_91d4_1655bd2d59e5.slice/crio-43f06a213d0227b461fd88a644e6a4e94795cfcac4f64ac3f599b2797924d7ea WatchSource:0}: Error finding container 43f06a213d0227b461fd88a644e6a4e94795cfcac4f64ac3f599b2797924d7ea: Status 404 returned error can't find the container with id 43f06a213d0227b461fd88a644e6a4e94795cfcac4f64ac3f599b2797924d7ea Nov 01 20:34:14.328565 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:14.328525 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" event={"ID":"2ce8176c-9f10-41d0-91d4-1655bd2d59e5","Type":"ContainerStarted","Data":"43f06a213d0227b461fd88a644e6a4e94795cfcac4f64ac3f599b2797924d7ea"} Nov 01 20:34:14.330491 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:14.330460 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" event={"ID":"84010e72-32ff-487e-ad03-7c6608757314","Type":"ContainerStarted","Data":"afc3604c862fc0a99c36f7b6c30139c3fab7f223245b394a3c49905a012f732c"} Nov 01 20:34:14.331781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:14.331748 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" event={"ID":"967c85c5-ef28-4ebc-bb49-62f0df3c23e9","Type":"ContainerStarted","Data":"97ecbb0446f29ae374a5c49a4f770ad4c4bff0aae0ce90b7e35e9b0e71c8b1ea"} Nov 01 20:34:14.332857 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:14.332828 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" event={"ID":"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3","Type":"ContainerStarted","Data":"1843f718fb8f6d1ec203106d1c3b7f9a008b766abb8798cdcf5f48e0268bb2f1"} Nov 01 20:34:15.056292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.056254 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr"] Nov 01 20:34:15.061097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.060875 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.156304 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.156265 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.156502 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.156330 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbl4\" (UniqueName: \"kubernetes.io/projected/11317270-0e35-445c-8ffb-ca52c7ecc452-kube-api-access-9wbl4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.156502 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.156394 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.166949 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.166913 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr"] Nov 01 20:34:15.257503 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.257458 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.257691 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.257554 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.257829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.257647 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbl4\" (UniqueName: \"kubernetes.io/projected/11317270-0e35-445c-8ffb-ca52c7ecc452-kube-api-access-9wbl4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.257891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.257877 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.258157 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.258133 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.275288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.273186 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5"] Nov 01 20:34:15.278723 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.278013 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.334452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.334420 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbl4\" (UniqueName: \"kubernetes.io/projected/11317270-0e35-445c-8ffb-ca52c7ecc452-kube-api-access-9wbl4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.335034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.335008 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5"] Nov 01 20:34:15.358705 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.358673 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.358892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.358736 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.358892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.358758 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqh74\" (UniqueName: \"kubernetes.io/projected/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-kube-api-access-hqh74\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.383241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.383206 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:15.460025 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.459994 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.460220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.460068 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.460220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.460122 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh74\" (UniqueName: \"kubernetes.io/projected/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-kube-api-access-hqh74\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.460802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.460780 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.460985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.460966 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.517446 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.517417 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqh74\" (UniqueName: \"kubernetes.io/projected/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-kube-api-access-hqh74\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.600496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.600181 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:15.819422 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:15.819221 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr"] Nov 01 20:34:15.823854 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:15.823821 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11317270_0e35_445c_8ffb_ca52c7ecc452.slice/crio-3a2207e07e8c647489937d8c66e44e27213d8ed54efb1f08e568b8efdbc659c1 WatchSource:0}: Error finding container 3a2207e07e8c647489937d8c66e44e27213d8ed54efb1f08e568b8efdbc659c1: Status 404 returned error can't find the container with id 3a2207e07e8c647489937d8c66e44e27213d8ed54efb1f08e568b8efdbc659c1 Nov 01 20:34:16.022440 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.022361 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5"] Nov 01 20:34:16.028742 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:16.028711 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4a0a6b_3415_4f6a_8bcb_fef9d9732f53.slice/crio-8e81545afff2b53615bdc2b8bf6f9466015e7a570255f759905ff97a7a2f28fe WatchSource:0}: Error finding container 8e81545afff2b53615bdc2b8bf6f9466015e7a570255f759905ff97a7a2f28fe: Status 404 returned error can't find the container with id 8e81545afff2b53615bdc2b8bf6f9466015e7a570255f759905ff97a7a2f28fe Nov 01 20:34:16.353636 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.353538 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" event={"ID":"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53","Type":"ContainerStarted","Data":"8e81545afff2b53615bdc2b8bf6f9466015e7a570255f759905ff97a7a2f28fe"} Nov 01 20:34:16.355424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.355349 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" event={"ID":"11317270-0e35-445c-8ffb-ca52c7ecc452","Type":"ContainerStarted","Data":"3a2207e07e8c647489937d8c66e44e27213d8ed54efb1f08e568b8efdbc659c1"} Nov 01 20:34:16.780683 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.780603 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj"] Nov 01 20:34:16.785116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.785066 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.857223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.856975 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 20:34:16.857223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.857053 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cqvhp\"" Nov 01 20:34:16.857223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.857141 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 20:34:16.857223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.857180 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876264 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbts\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-kube-api-access-tbbts\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876323 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876355 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876384 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876421 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d7cf7c-22e5-4201-852e-98b1b0944656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876469 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876506 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876533 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.876919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.876570 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.888060 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.887932 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj"] Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977170 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977231 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977260 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977292 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977336 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbts\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-kube-api-access-tbbts\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977390 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977419 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977448 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.977489 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d7cf7c-22e5-4201-852e-98b1b0944656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.978285 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d7cf7c-22e5-4201-852e-98b1b0944656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.978893 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.979135 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.980206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.979633 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.981403 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.981312 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.994122 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.993379 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:16.996986 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:16.996937 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:17.060030 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.059957 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:17.073639 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.073574 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbts\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-kube-api-access-tbbts\") pod \"maas-default-gateway-openshift-default-58c6585447-z2jrj\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:17.094445 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.093761 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq"] Nov 01 20:34:17.107443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.107212 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:17.111217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.111167 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.157713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.157666 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq"] Nov 01 20:34:17.169650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.164044 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-8qgkr\"" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179484 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179548 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e0759-b495-4e67-8bae-b96b8c77ef36-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179577 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnh2\" (UniqueName: \"kubernetes.io/projected/854e0759-b495-4e67-8bae-b96b8c77ef36-kube-api-access-7rnh2\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179624 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179655 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179680 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179721 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179766 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.179984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.179799 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281047 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281018 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281068 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e0759-b495-4e67-8bae-b96b8c77ef36-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281109 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnh2\" (UniqueName: \"kubernetes.io/projected/854e0759-b495-4e67-8bae-b96b8c77ef36-kube-api-access-7rnh2\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281137 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281162 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281179 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281205 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281238 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281619 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281259 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281619 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281590 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.281790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.281770 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.282280 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.282260 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e0759-b495-4e67-8bae-b96b8c77ef36-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.283003 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.282977 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.283118 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.283010 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.285692 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.285658 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.293497 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.293232 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.343028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.342996 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e0759-b495-4e67-8bae-b96b8c77ef36-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.346957 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.346922 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnh2\" (UniqueName: \"kubernetes.io/projected/854e0759-b495-4e67-8bae-b96b8c77ef36-kube-api-access-7rnh2\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq\" (UID: \"854e0759-b495-4e67-8bae-b96b8c77ef36\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.432127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.432086 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:17.649502 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.649418 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj"] Nov 01 20:34:17.888981 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:17.888837 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq"] Nov 01 20:34:17.895106 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:34:17.895055 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854e0759_b495_4e67_8bae_b96b8c77ef36.slice/crio-4783bf588842650e3c194e32ba2612a678032596493784e30ab61836d46a0f63 WatchSource:0}: Error finding container 4783bf588842650e3c194e32ba2612a678032596493784e30ab61836d46a0f63: Status 404 returned error can't find the container with id 4783bf588842650e3c194e32ba2612a678032596493784e30ab61836d46a0f63 Nov 01 20:34:18.396564 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:18.396474 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" event={"ID":"80d7cf7c-22e5-4201-852e-98b1b0944656","Type":"ContainerStarted","Data":"a54f2ad6c705ed2d5d6c325eecc40835c7616771a0371ed5dc432af4004653fc"} Nov 01 20:34:18.400698 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:18.400667 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" event={"ID":"854e0759-b495-4e67-8bae-b96b8c77ef36","Type":"ContainerStarted","Data":"4783bf588842650e3c194e32ba2612a678032596493784e30ab61836d46a0f63"} Nov 01 20:34:28.842269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:28.842206 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:34:28.842634 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:28.842306 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:34:28.842634 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:28.842340 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:34:28.971310 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:28.971265 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:34:28.971463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:28.971346 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:34:28.971463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:28.971385 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:34:29.512344 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.512304 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" event={"ID":"854e0759-b495-4e67-8bae-b96b8c77ef36","Type":"ContainerStarted","Data":"740539d1f865e1621918d861b65cddc1628d1a3300f73273aa343347370c563d"} Nov 01 20:34:29.515277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.515245 4477 generic.go:356] "Generic (PLEG): container finished" podID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerID="10f537a7ecb34900ad364f62e274b9ba62bca6dbfa66fa32422f3cb9d20091a8" exitCode=0 Nov 01 20:34:29.515432 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.515334 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" event={"ID":"3a978b7c-ca9f-4391-b26d-53ba29f18c2e","Type":"ContainerDied","Data":"10f537a7ecb34900ad364f62e274b9ba62bca6dbfa66fa32422f3cb9d20091a8"} Nov 01 20:34:29.518899 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.518877 4477 generic.go:356] "Generic (PLEG): container finished" podID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerID="ebc6e3bcd50f2b4ae83b8fe9d9e4d9db5822a0105a5bf31e693dfff178e8e4db" exitCode=0 Nov 01 20:34:29.518984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.518962 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" event={"ID":"8b16c78d-103c-438e-ad5d-103fc6e68ac2","Type":"ContainerDied","Data":"ebc6e3bcd50f2b4ae83b8fe9d9e4d9db5822a0105a5bf31e693dfff178e8e4db"} Nov 01 20:34:29.523832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.523337 4477 generic.go:356] "Generic (PLEG): container finished" podID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerID="d2a4adc2d340dc25ac98b2181043cc0d9b864cc03a111dcf6b5ca7575f1adde3" exitCode=0 Nov 01 20:34:29.523832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.523409 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" event={"ID":"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53","Type":"ContainerDied","Data":"d2a4adc2d340dc25ac98b2181043cc0d9b864cc03a111dcf6b5ca7575f1adde3"} Nov 01 20:34:29.527910 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.527883 4477 generic.go:356] "Generic (PLEG): container finished" podID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerID="c63eb7b11430cf3aab664ef08edcfeda226a1c1243c54d550a159b919687e3a1" exitCode=0 Nov 01 20:34:29.528066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.528006 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" event={"ID":"11317270-0e35-445c-8ffb-ca52c7ecc452","Type":"ContainerDied","Data":"c63eb7b11430cf3aab664ef08edcfeda226a1c1243c54d550a159b919687e3a1"} Nov 01 20:34:29.531349 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.531319 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" event={"ID":"80d7cf7c-22e5-4201-852e-98b1b0944656","Type":"ContainerStarted","Data":"1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e"} Nov 01 20:34:29.721156 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.721031 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" podStartSLOduration=1.7815890909999998 podStartE2EDuration="12.721013474s" totalImagesPullingTime="10.939424383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:29.67766113 +0000 UTC m=+223.327712736" watchObservedRunningTime="2026-11-01 20:34:29.721013474 +0000 UTC m=+223.371065021" Nov 01 20:34:29.914910 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:29.914326 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" podStartSLOduration=2.615839007 podStartE2EDuration="13.914304798s" totalImagesPullingTime="11.298465791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:29.91222006 +0000 UTC m=+223.562271622" watchObservedRunningTime="2026-11-01 20:34:29.914304798 +0000 UTC m=+223.564356341" Nov 01 20:34:30.108504 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:30.108465 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:30.114192 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:30.114120 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:30.433583 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:30.433506 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:30.439271 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:30.439244 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:30.537295 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:30.537271 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:30.537658 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:30.537570 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:30.538133 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:30.538110 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq" Nov 01 20:34:30.538741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:30.538723 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:34:31.545865 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:31.545728 4477 generic.go:356] "Generic (PLEG): container finished" podID="12b6dd46-9985-44f9-a905-d904087774d1" containerID="65c3d3fe4f468d32175407efa61ad5dc7d9759077edcf674d6e72b47eac7d53b" exitCode=0 Nov 01 20:34:31.547637 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:31.546294 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" event={"ID":"12b6dd46-9985-44f9-a905-d904087774d1","Type":"ContainerDied","Data":"65c3d3fe4f468d32175407efa61ad5dc7d9759077edcf674d6e72b47eac7d53b"} Nov 01 20:34:32.554387 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:32.554353 4477 generic.go:356] "Generic (PLEG): container finished" podID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerID="417cebc961cb06b87fc6a0e6f5eb0558a5d851f3a96b6bc5508d7c65e5892104" exitCode=0 Nov 01 20:34:32.554845 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:32.554462 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" event={"ID":"2ce8176c-9f10-41d0-91d4-1655bd2d59e5","Type":"ContainerDied","Data":"417cebc961cb06b87fc6a0e6f5eb0558a5d851f3a96b6bc5508d7c65e5892104"} Nov 01 20:34:32.558367 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:32.558338 4477 generic.go:356] "Generic (PLEG): container finished" podID="84010e72-32ff-487e-ad03-7c6608757314" containerID="e850fac34aad76b312425f82e6f77ac6d860de6847586fa78115e22a8545f4bd" exitCode=0 Nov 01 20:34:32.558478 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:32.558438 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" event={"ID":"84010e72-32ff-487e-ad03-7c6608757314","Type":"ContainerDied","Data":"e850fac34aad76b312425f82e6f77ac6d860de6847586fa78115e22a8545f4bd"} Nov 01 20:34:32.562177 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:32.562126 4477 generic.go:356] "Generic (PLEG): container finished" podID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerID="d6ec3c3f80a77f7bfb17b7e1866de32cb6d1d1e9b4a258e7e92b71e0154d76d1" exitCode=0 Nov 01 20:34:32.562274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:32.562253 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" event={"ID":"967c85c5-ef28-4ebc-bb49-62f0df3c23e9","Type":"ContainerDied","Data":"d6ec3c3f80a77f7bfb17b7e1866de32cb6d1d1e9b4a258e7e92b71e0154d76d1"} Nov 01 20:34:32.564836 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:32.564810 4477 generic.go:356] "Generic (PLEG): container finished" podID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerID="1c13e2fd44f620801b55e6182d6c413c7937d0eb33cb96332f947f2e3ee29856" exitCode=0 Nov 01 20:34:32.565858 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:32.565835 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" event={"ID":"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3","Type":"ContainerDied","Data":"1c13e2fd44f620801b55e6182d6c413c7937d0eb33cb96332f947f2e3ee29856"} Nov 01 20:34:33.574383 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:33.573583 4477 generic.go:356] "Generic (PLEG): container finished" podID="12b6dd46-9985-44f9-a905-d904087774d1" containerID="aef510e2e26240904d847ee57eb9e5084f487cb1904bd8fbbc817cdc6913c200" exitCode=0 Nov 01 20:34:33.574383 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:33.573715 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" event={"ID":"12b6dd46-9985-44f9-a905-d904087774d1","Type":"ContainerDied","Data":"aef510e2e26240904d847ee57eb9e5084f487cb1904bd8fbbc817cdc6913c200"} Nov 01 20:34:34.582989 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.582955 4477 generic.go:356] "Generic (PLEG): container finished" podID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerID="7ddd68b1558b65840dfa8daeab614f08c518e977a433b49489b3225432654d2c" exitCode=0 Nov 01 20:34:34.583454 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.583037 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" event={"ID":"11317270-0e35-445c-8ffb-ca52c7ecc452","Type":"ContainerDied","Data":"7ddd68b1558b65840dfa8daeab614f08c518e977a433b49489b3225432654d2c"} Nov 01 20:34:34.586673 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.586561 4477 generic.go:356] "Generic (PLEG): container finished" podID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerID="7b716ba180530e4107ed198ac85f89f88b59b165cd117a8f002f198044dfa279" exitCode=0 Nov 01 20:34:34.586673 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.586610 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" event={"ID":"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3","Type":"ContainerDied","Data":"7b716ba180530e4107ed198ac85f89f88b59b165cd117a8f002f198044dfa279"} Nov 01 20:34:34.589412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.589305 4477 generic.go:356] "Generic (PLEG): container finished" podID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerID="d55dc5ee89532f1577ae047084afc4de7efd6a3b6cddf0f7f429da4a469741a5" exitCode=0 Nov 01 20:34:34.589412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.589394 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" event={"ID":"3a978b7c-ca9f-4391-b26d-53ba29f18c2e","Type":"ContainerDied","Data":"d55dc5ee89532f1577ae047084afc4de7efd6a3b6cddf0f7f429da4a469741a5"} Nov 01 20:34:34.593044 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.593016 4477 generic.go:356] "Generic (PLEG): container finished" podID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerID="1eac40b6425a94ab8da1b2dd7d5826adbdce372558ec454413b182b577026ea9" exitCode=0 Nov 01 20:34:34.593145 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.593110 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" event={"ID":"8b16c78d-103c-438e-ad5d-103fc6e68ac2","Type":"ContainerDied","Data":"1eac40b6425a94ab8da1b2dd7d5826adbdce372558ec454413b182b577026ea9"} Nov 01 20:34:34.596115 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.596069 4477 generic.go:356] "Generic (PLEG): container finished" podID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerID="f157b08baad4de20af56f0a2ceecd65d0bb0eb7f8427c5bb54545e494dd0dd2b" exitCode=0 Nov 01 20:34:34.596210 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:34.596142 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" event={"ID":"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53","Type":"ContainerDied","Data":"f157b08baad4de20af56f0a2ceecd65d0bb0eb7f8427c5bb54545e494dd0dd2b"} Nov 01 20:34:35.607088 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:35.605953 4477 generic.go:356] "Generic (PLEG): container finished" podID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerID="f8c6a486bb6887e1a13f406c08b9336ea57b0ae480b492bcbc0fc14952d60fe4" exitCode=0 Nov 01 20:34:35.607088 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:35.606049 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" event={"ID":"2ce8176c-9f10-41d0-91d4-1655bd2d59e5","Type":"ContainerDied","Data":"f8c6a486bb6887e1a13f406c08b9336ea57b0ae480b492bcbc0fc14952d60fe4"} Nov 01 20:34:35.613798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:35.613005 4477 generic.go:356] "Generic (PLEG): container finished" podID="84010e72-32ff-487e-ad03-7c6608757314" containerID="381c7332784959bd00bc8688af00486b56ef915318987f689c148e866ef0cb2a" exitCode=0 Nov 01 20:34:35.613798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:35.613096 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" event={"ID":"84010e72-32ff-487e-ad03-7c6608757314","Type":"ContainerDied","Data":"381c7332784959bd00bc8688af00486b56ef915318987f689c148e866ef0cb2a"} Nov 01 20:34:36.570856 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:34:36.570811 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:34:37.636691 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:37.636650 4477 generic.go:356] "Generic (PLEG): container finished" podID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerID="b6a0987f1fbe9879b2bd24e867bcfa6314638cb8055d0f46b4e8e9e018ec4d5d" exitCode=0 Nov 01 20:34:37.637068 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:37.636731 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" event={"ID":"967c85c5-ef28-4ebc-bb49-62f0df3c23e9","Type":"ContainerDied","Data":"b6a0987f1fbe9879b2bd24e867bcfa6314638cb8055d0f46b4e8e9e018ec4d5d"} Nov 01 20:34:52.830173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:52.829547 4477 generic.go:356] "Generic (PLEG): container finished" podID="12b6dd46-9985-44f9-a905-d904087774d1" containerID="bd2a24dcfeec1977dc21261fc1038d93783a837f0cfb87f69ecfe904f7ca30a6" exitCode=0 Nov 01 20:34:52.830173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:52.829688 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" event={"ID":"12b6dd46-9985-44f9-a905-d904087774d1","Type":"ContainerDied","Data":"bd2a24dcfeec1977dc21261fc1038d93783a837f0cfb87f69ecfe904f7ca30a6"} Nov 01 20:34:53.840923 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:53.840891 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" event={"ID":"8b16c78d-103c-438e-ad5d-103fc6e68ac2","Type":"ContainerStarted","Data":"d023cfb3fdd5dcf234792c086a5054cde40ed125831f46ace343033dc079d0e2"} Nov 01 20:34:53.845178 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:53.845148 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" event={"ID":"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53","Type":"ContainerStarted","Data":"381e7046f48020d5e3aea0e8c3995aee9a973b9b91e2303d74c3a897913f6da9"} Nov 01 20:34:53.848615 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:53.848583 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" event={"ID":"11317270-0e35-445c-8ffb-ca52c7ecc452","Type":"ContainerStarted","Data":"d1de39ca94b1af667af6e7667e18d4e4ba39c77df1fdf95270b47ee02f98ebdb"} Nov 01 20:34:53.998357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:53.997995 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" podStartSLOduration=3.768613141 podStartE2EDuration="45.997972272s" totalImagesPullingTime="42.229359131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:53.996540185 +0000 UTC m=+247.646591750" watchObservedRunningTime="2026-11-01 20:34:53.997972272 +0000 UTC m=+247.648023819" Nov 01 20:34:54.022424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.022392 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:54.112732 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.112683 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" podStartSLOduration=2.964473271 podStartE2EDuration="39.112666988s" totalImagesPullingTime="36.148193717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:54.110491711 +0000 UTC m=+247.760543257" watchObservedRunningTime="2026-11-01 20:34:54.112666988 +0000 UTC m=+247.762718566" Nov 01 20:34:54.176103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.171790 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-util\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-util\") pod \"12b6dd46-9985-44f9-a905-d904087774d1\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " Nov 01 20:34:54.176103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.171838 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-bundle\") pod \"12b6dd46-9985-44f9-a905-d904087774d1\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " Nov 01 20:34:54.176103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.171938 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b6dd46-9985-44f9-a905-d904087774d1-kube-api-access-dbnf9\" (UniqueName: \"kubernetes.io/projected/12b6dd46-9985-44f9-a905-d904087774d1-kube-api-access-dbnf9\") pod \"12b6dd46-9985-44f9-a905-d904087774d1\" (UID: \"12b6dd46-9985-44f9-a905-d904087774d1\") " Nov 01 20:34:54.177521 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.177486 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-bundle" pod "12b6dd46-9985-44f9-a905-d904087774d1" (UID: "12b6dd46-9985-44f9-a905-d904087774d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:54.177793 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.177700 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b6dd46-9985-44f9-a905-d904087774d1-kube-api-access-dbnf9" pod "12b6dd46-9985-44f9-a905-d904087774d1" (UID: "12b6dd46-9985-44f9-a905-d904087774d1"). InnerVolumeSpecName "kube-api-access-dbnf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:54.184344 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.184318 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-util" pod "12b6dd46-9985-44f9-a905-d904087774d1" (UID: "12b6dd46-9985-44f9-a905-d904087774d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:54.230802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.230651 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" podStartSLOduration=2.930325222 podStartE2EDuration="39.230636339s" totalImagesPullingTime="36.300311117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:54.230277358 +0000 UTC m=+247.880328905" watchObservedRunningTime="2026-11-01 20:34:54.230636339 +0000 UTC m=+247.880687885" Nov 01 20:34:54.273232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.273190 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbnf9\" (UniqueName: \"kubernetes.io/projected/12b6dd46-9985-44f9-a905-d904087774d1-kube-api-access-dbnf9\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:54.273232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.273224 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:54.273232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.273239 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b6dd46-9985-44f9-a905-d904087774d1-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:54.856805 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.856777 4477 generic.go:356] "Generic (PLEG): container finished" podID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerID="90ef37bec1d61854460cc67d8294c4febee3a10722997ed988ccf694e63c4261" exitCode=0 Nov 01 20:34:54.857237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.856859 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" event={"ID":"3a978b7c-ca9f-4391-b26d-53ba29f18c2e","Type":"ContainerDied","Data":"90ef37bec1d61854460cc67d8294c4febee3a10722997ed988ccf694e63c4261"} Nov 01 20:34:54.859601 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.859576 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" event={"ID":"2ce8176c-9f10-41d0-91d4-1655bd2d59e5","Type":"ContainerStarted","Data":"f1cdfbb490a966f981b98ae5bd7044a2689184b42fe8e0dd34b1ce77cfead132"} Nov 01 20:34:54.862131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.862110 4477 generic.go:356] "Generic (PLEG): container finished" podID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerID="d023cfb3fdd5dcf234792c086a5054cde40ed125831f46ace343033dc079d0e2" exitCode=0 Nov 01 20:34:54.862306 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.862288 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" event={"ID":"8b16c78d-103c-438e-ad5d-103fc6e68ac2","Type":"ContainerDied","Data":"d023cfb3fdd5dcf234792c086a5054cde40ed125831f46ace343033dc079d0e2"} Nov 01 20:34:54.864772 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.864750 4477 generic.go:356] "Generic (PLEG): container finished" podID="84010e72-32ff-487e-ad03-7c6608757314" containerID="e3c1a54793571b4cb6c4a7033f715708f09a33c38a067564711d61f8b0f8a820" exitCode=0 Nov 01 20:34:54.864863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.864823 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" event={"ID":"84010e72-32ff-487e-ad03-7c6608757314","Type":"ContainerDied","Data":"e3c1a54793571b4cb6c4a7033f715708f09a33c38a067564711d61f8b0f8a820"} Nov 01 20:34:54.867415 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.867390 4477 generic.go:356] "Generic (PLEG): container finished" podID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerID="381e7046f48020d5e3aea0e8c3995aee9a973b9b91e2303d74c3a897913f6da9" exitCode=0 Nov 01 20:34:54.867550 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.867474 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" event={"ID":"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53","Type":"ContainerDied","Data":"381e7046f48020d5e3aea0e8c3995aee9a973b9b91e2303d74c3a897913f6da9"} Nov 01 20:34:54.869679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.869660 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" Nov 01 20:34:54.869679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.869671 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l688c" event={"ID":"12b6dd46-9985-44f9-a905-d904087774d1","Type":"ContainerDied","Data":"7251af55f1a848ca54245f11ab05d80bc1fc52c482021869f4eb233e80c20be5"} Nov 01 20:34:54.869856 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.869708 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7251af55f1a848ca54245f11ab05d80bc1fc52c482021869f4eb233e80c20be5" Nov 01 20:34:54.872043 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.872013 4477 generic.go:356] "Generic (PLEG): container finished" podID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerID="d1de39ca94b1af667af6e7667e18d4e4ba39c77df1fdf95270b47ee02f98ebdb" exitCode=0 Nov 01 20:34:54.872143 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.872130 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" event={"ID":"11317270-0e35-445c-8ffb-ca52c7ecc452","Type":"ContainerDied","Data":"d1de39ca94b1af667af6e7667e18d4e4ba39c77df1fdf95270b47ee02f98ebdb"} Nov 01 20:34:54.874536 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.874501 4477 generic.go:356] "Generic (PLEG): container finished" podID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerID="3c9adbe9e5b7b52f276e341c49996daca5f6fc4b80e3eaa39b953b941801e362" exitCode=0 Nov 01 20:34:54.874608 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:54.874548 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" event={"ID":"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3","Type":"ContainerDied","Data":"3c9adbe9e5b7b52f276e341c49996daca5f6fc4b80e3eaa39b953b941801e362"} Nov 01 20:34:55.428476 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:55.428015 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" podStartSLOduration=3.575923648 podStartE2EDuration="42.427995297s" totalImagesPullingTime="38.852071649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:55.426800996 +0000 UTC m=+249.076852541" watchObservedRunningTime="2026-11-01 20:34:55.427995297 +0000 UTC m=+249.078046844" Nov 01 20:34:55.880815 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:55.880777 4477 generic.go:356] "Generic (PLEG): container finished" podID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerID="f1cdfbb490a966f981b98ae5bd7044a2689184b42fe8e0dd34b1ce77cfead132" exitCode=0 Nov 01 20:34:55.881239 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:55.880860 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" event={"ID":"2ce8176c-9f10-41d0-91d4-1655bd2d59e5","Type":"ContainerDied","Data":"f1cdfbb490a966f981b98ae5bd7044a2689184b42fe8e0dd34b1ce77cfead132"} Nov 01 20:34:55.883151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:55.883129 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" event={"ID":"967c85c5-ef28-4ebc-bb49-62f0df3c23e9","Type":"ContainerStarted","Data":"351a47f797d2bd2be2a8a02c3a15b6efe1d482efea3eb5d68091ed0a6ea641ad"} Nov 01 20:34:56.094978 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.094950 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:56.107108 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.103265 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:56.112305 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.112245 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" podStartSLOduration=3.782952475 podStartE2EDuration="44.11222313s" totalImagesPullingTime="40.329270655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:56.109303232 +0000 UTC m=+249.759354780" watchObservedRunningTime="2026-11-01 20:34:56.11222313 +0000 UTC m=+249.762274676" Nov 01 20:34:56.141394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.141345 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:56.144242 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.144224 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:56.147753 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.147734 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:56.150629 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.150613 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:56.192988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.192970 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-util\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-util\") pod \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " Nov 01 20:34:56.193098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.193012 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-kube-api-access-hqh74\" (UniqueName: \"kubernetes.io/projected/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-kube-api-access-hqh74\") pod \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " Nov 01 20:34:56.193098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.193053 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-bundle\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-bundle\") pod \"84010e72-32ff-487e-ad03-7c6608757314\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " Nov 01 20:34:56.193098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.193069 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-bundle\") pod \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\" (UID: \"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53\") " Nov 01 20:34:56.193217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.193112 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-util\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-util\") pod \"84010e72-32ff-487e-ad03-7c6608757314\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " Nov 01 20:34:56.193217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.193131 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84010e72-32ff-487e-ad03-7c6608757314-kube-api-access-8xk4d\" (UniqueName: \"kubernetes.io/projected/84010e72-32ff-487e-ad03-7c6608757314-kube-api-access-8xk4d\") pod \"84010e72-32ff-487e-ad03-7c6608757314\" (UID: \"84010e72-32ff-487e-ad03-7c6608757314\") " Nov 01 20:34:56.193823 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.193777 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-bundle" pod "6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" (UID: "6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.193905 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.193829 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-bundle" pod "84010e72-32ff-487e-ad03-7c6608757314" (UID: "84010e72-32ff-487e-ad03-7c6608757314"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.195333 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.195313 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84010e72-32ff-487e-ad03-7c6608757314-kube-api-access-8xk4d" pod "84010e72-32ff-487e-ad03-7c6608757314" (UID: "84010e72-32ff-487e-ad03-7c6608757314"). InnerVolumeSpecName "kube-api-access-8xk4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:56.195624 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.195600 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-kube-api-access-hqh74" pod "6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" (UID: "6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53"). InnerVolumeSpecName "kube-api-access-hqh74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:56.198850 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.198830 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-util" pod "84010e72-32ff-487e-ad03-7c6608757314" (UID: "84010e72-32ff-487e-ad03-7c6608757314"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.198902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.198849 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-util" pod "6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" (UID: "6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.293852 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.293819 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-util\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-util\") pod \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " Nov 01 20:34:56.294018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.293868 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-bundle\") pod \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " Nov 01 20:34:56.294018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.293889 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11317270-0e35-445c-8ffb-ca52c7ecc452-kube-api-access-9wbl4\" (UniqueName: \"kubernetes.io/projected/11317270-0e35-445c-8ffb-ca52c7ecc452-kube-api-access-9wbl4\") pod \"11317270-0e35-445c-8ffb-ca52c7ecc452\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " Nov 01 20:34:56.294018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.293914 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-bundle\") pod \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " Nov 01 20:34:56.294018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.293936 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-util\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-util\") pod \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " Nov 01 20:34:56.294018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.293953 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-kube-api-access-sc22t\" (UniqueName: \"kubernetes.io/projected/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-kube-api-access-sc22t\") pod \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " Nov 01 20:34:56.294018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.293971 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-util\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-util\") pod \"11317270-0e35-445c-8ffb-ca52c7ecc452\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " Nov 01 20:34:56.294018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.294012 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b16c78d-103c-438e-ad5d-103fc6e68ac2-kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/8b16c78d-103c-438e-ad5d-103fc6e68ac2-kube-api-access-gjrb7\") pod \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\" (UID: \"8b16c78d-103c-438e-ad5d-103fc6e68ac2\") " Nov 01 20:34:56.294443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.294142 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-bundle\") pod \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " Nov 01 20:34:56.294443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.294187 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-kube-api-access-vnfkz\" (UniqueName: \"kubernetes.io/projected/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-kube-api-access-vnfkz\") pod \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\" (UID: \"3a978b7c-ca9f-4391-b26d-53ba29f18c2e\") " Nov 01 20:34:56.294443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.294213 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-bundle\") pod \"11317270-0e35-445c-8ffb-ca52c7ecc452\" (UID: \"11317270-0e35-445c-8ffb-ca52c7ecc452\") " Nov 01 20:34:56.294443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.294268 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-util\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-util\") pod \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\" (UID: \"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3\") " Nov 01 20:34:56.294443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.294393 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-bundle" pod "8b16c78d-103c-438e-ad5d-103fc6e68ac2" (UID: "8b16c78d-103c-438e-ad5d-103fc6e68ac2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.296847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.295575 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.296847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.295604 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqh74\" (UniqueName: \"kubernetes.io/projected/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-kube-api-access-hqh74\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.296847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.295620 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.296847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.295635 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.296847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.295653 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.296847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.294465 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-bundle" pod "4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" (UID: "4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.297637 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.297534 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-bundle" pod "3a978b7c-ca9f-4391-b26d-53ba29f18c2e" (UID: "3a978b7c-ca9f-4391-b26d-53ba29f18c2e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.297827 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.297799 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-bundle" pod "11317270-0e35-445c-8ffb-ca52c7ecc452" (UID: "11317270-0e35-445c-8ffb-ca52c7ecc452"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.297923 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.297907 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84010e72-32ff-487e-ad03-7c6608757314-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.297985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.297947 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xk4d\" (UniqueName: \"kubernetes.io/projected/84010e72-32ff-487e-ad03-7c6608757314-kube-api-access-8xk4d\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.303593 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.302458 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-util" pod "3a978b7c-ca9f-4391-b26d-53ba29f18c2e" (UID: "3a978b7c-ca9f-4391-b26d-53ba29f18c2e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.303593 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.302490 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-util" pod "8b16c78d-103c-438e-ad5d-103fc6e68ac2" (UID: "8b16c78d-103c-438e-ad5d-103fc6e68ac2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.303755 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.303727 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11317270-0e35-445c-8ffb-ca52c7ecc452-kube-api-access-9wbl4" pod "11317270-0e35-445c-8ffb-ca52c7ecc452" (UID: "11317270-0e35-445c-8ffb-ca52c7ecc452"). InnerVolumeSpecName "kube-api-access-9wbl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:56.304405 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.304373 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b16c78d-103c-438e-ad5d-103fc6e68ac2-kube-api-access-gjrb7" pod "8b16c78d-103c-438e-ad5d-103fc6e68ac2" (UID: "8b16c78d-103c-438e-ad5d-103fc6e68ac2"). InnerVolumeSpecName "kube-api-access-gjrb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:56.304518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.304411 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-kube-api-access-sc22t" pod "4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" (UID: "4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3"). InnerVolumeSpecName "kube-api-access-sc22t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:56.304518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.304424 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-kube-api-access-vnfkz" pod "3a978b7c-ca9f-4391-b26d-53ba29f18c2e" (UID: "3a978b7c-ca9f-4391-b26d-53ba29f18c2e"). InnerVolumeSpecName "kube-api-access-vnfkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:56.305857 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.305837 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-util" pod "4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" (UID: "4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.309269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.309235 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-util" pod "11317270-0e35-445c-8ffb-ca52c7ecc452" (UID: "11317270-0e35-445c-8ffb-ca52c7ecc452"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398481 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wbl4\" (UniqueName: \"kubernetes.io/projected/11317270-0e35-445c-8ffb-ca52c7ecc452-kube-api-access-9wbl4\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398523 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398533 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398541 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc22t\" (UniqueName: \"kubernetes.io/projected/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-kube-api-access-sc22t\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398551 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398559 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/8b16c78d-103c-438e-ad5d-103fc6e68ac2-kube-api-access-gjrb7\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398567 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398576 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnfkz\" (UniqueName: \"kubernetes.io/projected/3a978b7c-ca9f-4391-b26d-53ba29f18c2e-kube-api-access-vnfkz\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398584 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11317270-0e35-445c-8ffb-ca52c7ecc452-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398591 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.398581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.398599 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b16c78d-103c-438e-ad5d-103fc6e68ac2-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:56.889982 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.889948 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" event={"ID":"3a978b7c-ca9f-4391-b26d-53ba29f18c2e","Type":"ContainerDied","Data":"d269b9e6bd3fd0b43f2c3a73fee5ef6c0a631c80eed11fdf480d87fb2b4288cf"} Nov 01 20:34:56.889982 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.889980 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpd9s" Nov 01 20:34:56.890528 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.890000 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d269b9e6bd3fd0b43f2c3a73fee5ef6c0a631c80eed11fdf480d87fb2b4288cf" Nov 01 20:34:56.891902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.891881 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" event={"ID":"8b16c78d-103c-438e-ad5d-103fc6e68ac2","Type":"ContainerDied","Data":"fa13da994877af4540d0821f4db60aabc9f716d97c9d3fbcd484545144e6dd64"} Nov 01 20:34:56.892119 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.891916 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m7zrk" Nov 01 20:34:56.892119 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.891921 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa13da994877af4540d0821f4db60aabc9f716d97c9d3fbcd484545144e6dd64" Nov 01 20:34:56.894150 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.894126 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" event={"ID":"84010e72-32ff-487e-ad03-7c6608757314","Type":"ContainerDied","Data":"afc3604c862fc0a99c36f7b6c30139c3fab7f223245b394a3c49905a012f732c"} Nov 01 20:34:56.894241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.894152 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4qn8" Nov 01 20:34:56.894241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.894212 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc3604c862fc0a99c36f7b6c30139c3fab7f223245b394a3c49905a012f732c" Nov 01 20:34:56.896145 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.896124 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" event={"ID":"6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53","Type":"ContainerDied","Data":"8e81545afff2b53615bdc2b8bf6f9466015e7a570255f759905ff97a7a2f28fe"} Nov 01 20:34:56.896316 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.896294 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e81545afff2b53615bdc2b8bf6f9466015e7a570255f759905ff97a7a2f28fe" Nov 01 20:34:56.896316 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.896305 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chhvn5" Nov 01 20:34:56.898518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.898496 4477 generic.go:356] "Generic (PLEG): container finished" podID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerID="351a47f797d2bd2be2a8a02c3a15b6efe1d482efea3eb5d68091ed0a6ea641ad" exitCode=0 Nov 01 20:34:56.898612 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.898561 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" event={"ID":"967c85c5-ef28-4ebc-bb49-62f0df3c23e9","Type":"ContainerDied","Data":"351a47f797d2bd2be2a8a02c3a15b6efe1d482efea3eb5d68091ed0a6ea641ad"} Nov 01 20:34:56.900432 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.900416 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" Nov 01 20:34:56.900522 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.900431 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447xksr" event={"ID":"11317270-0e35-445c-8ffb-ca52c7ecc452","Type":"ContainerDied","Data":"3a2207e07e8c647489937d8c66e44e27213d8ed54efb1f08e568b8efdbc659c1"} Nov 01 20:34:56.900522 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.900466 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a2207e07e8c647489937d8c66e44e27213d8ed54efb1f08e568b8efdbc659c1" Nov 01 20:34:56.902526 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.902497 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" event={"ID":"4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3","Type":"ContainerDied","Data":"1843f718fb8f6d1ec203106d1c3b7f9a008b766abb8798cdcf5f48e0268bb2f1"} Nov 01 20:34:56.902597 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.902537 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1843f718fb8f6d1ec203106d1c3b7f9a008b766abb8798cdcf5f48e0268bb2f1" Nov 01 20:34:56.902774 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:56.902758 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6klp67" Nov 01 20:34:57.025084 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.025051 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:57.204529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.204454 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-util\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-util\") pod \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " Nov 01 20:34:57.204529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.204512 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-bundle\") pod \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " Nov 01 20:34:57.204724 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.204536 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-kube-api-access-mj7tt\" (UniqueName: \"kubernetes.io/projected/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-kube-api-access-mj7tt\") pod \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\" (UID: \"2ce8176c-9f10-41d0-91d4-1655bd2d59e5\") " Nov 01 20:34:57.205050 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.205022 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-bundle" pod "2ce8176c-9f10-41d0-91d4-1655bd2d59e5" (UID: "2ce8176c-9f10-41d0-91d4-1655bd2d59e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:57.206783 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.206757 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-kube-api-access-mj7tt" pod "2ce8176c-9f10-41d0-91d4-1655bd2d59e5" (UID: "2ce8176c-9f10-41d0-91d4-1655bd2d59e5"). InnerVolumeSpecName "kube-api-access-mj7tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:57.212346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.212320 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-util" pod "2ce8176c-9f10-41d0-91d4-1655bd2d59e5" (UID: "2ce8176c-9f10-41d0-91d4-1655bd2d59e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:57.305244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.305204 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:57.305244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.305234 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:57.305244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.305243 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj7tt\" (UniqueName: \"kubernetes.io/projected/2ce8176c-9f10-41d0-91d4-1655bd2d59e5-kube-api-access-mj7tt\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:57.909162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.909120 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" event={"ID":"2ce8176c-9f10-41d0-91d4-1655bd2d59e5","Type":"ContainerDied","Data":"43f06a213d0227b461fd88a644e6a4e94795cfcac4f64ac3f599b2797924d7ea"} Nov 01 20:34:57.909162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.909152 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tzs5" Nov 01 20:34:57.909583 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:57.909190 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f06a213d0227b461fd88a644e6a4e94795cfcac4f64ac3f599b2797924d7ea" Nov 01 20:34:58.029910 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.029891 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:58.213250 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.213162 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-bundle\") pod \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " Nov 01 20:34:58.213408 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.213286 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-util\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-util\") pod \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " Nov 01 20:34:58.213408 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.213322 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-kube-api-access-nt9p6\" (UniqueName: \"kubernetes.io/projected/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-kube-api-access-nt9p6\") pod \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\" (UID: \"967c85c5-ef28-4ebc-bb49-62f0df3c23e9\") " Nov 01 20:34:58.213737 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.213710 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-bundle" pod "967c85c5-ef28-4ebc-bb49-62f0df3c23e9" (UID: "967c85c5-ef28-4ebc-bb49-62f0df3c23e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:58.215663 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.215641 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-kube-api-access-nt9p6" pod "967c85c5-ef28-4ebc-bb49-62f0df3c23e9" (UID: "967c85c5-ef28-4ebc-bb49-62f0df3c23e9"). InnerVolumeSpecName "kube-api-access-nt9p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:58.219060 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.219036 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-util" pod "967c85c5-ef28-4ebc-bb49-62f0df3c23e9" (UID: "967c85c5-ef28-4ebc-bb49-62f0df3c23e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:58.314267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.314227 4477 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-util\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:58.314267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.314263 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt9p6\" (UniqueName: \"kubernetes.io/projected/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-kube-api-access-nt9p6\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:58.314267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.314276 4477 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967c85c5-ef28-4ebc-bb49-62f0df3c23e9-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:58.915117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.915086 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" Nov 01 20:34:58.915117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.915100 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjs5sc" event={"ID":"967c85c5-ef28-4ebc-bb49-62f0df3c23e9","Type":"ContainerDied","Data":"97ecbb0446f29ae374a5c49a4f770ad4c4bff0aae0ce90b7e35e9b0e71c8b1ea"} Nov 01 20:34:58.915601 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:34:58.915147 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ecbb0446f29ae374a5c49a4f770ad4c4bff0aae0ce90b7e35e9b0e71c8b1ea" Nov 01 20:35:00.126157 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126123 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v"] Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126396 4477 cpu_manager.go:414] "containerMap: removing container" podUID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126406 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126416 4477 cpu_manager.go:414] "containerMap: removing container" podUID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126420 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126426 4477 cpu_manager.go:414] "containerMap: removing container" podUID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126431 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126439 4477 cpu_manager.go:414] "containerMap: removing container" podUID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126443 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126449 4477 cpu_manager.go:414] "containerMap: removing container" podUID="84010e72-32ff-487e-ad03-7c6608757314" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126454 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84010e72-32ff-487e-ad03-7c6608757314" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126461 4477 cpu_manager.go:414] "containerMap: removing container" podUID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerName="pull" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126466 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerName="pull" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126474 4477 cpu_manager.go:414] "containerMap: removing container" podUID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerName="pull" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126493 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerName="pull" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126499 4477 cpu_manager.go:414] "containerMap: removing container" podUID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerName="pull" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126504 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerName="pull" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126512 4477 cpu_manager.go:414] "containerMap: removing container" podUID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerName="extract" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126516 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerName="extract" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126522 4477 cpu_manager.go:414] "containerMap: removing container" podUID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerName="extract" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126526 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerName="extract" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126537 4477 cpu_manager.go:414] "containerMap: removing container" podUID="84010e72-32ff-487e-ad03-7c6608757314" containerName="extract" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126541 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84010e72-32ff-487e-ad03-7c6608757314" containerName="extract" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126549 4477 cpu_manager.go:414] "containerMap: removing container" podUID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerName="util" Nov 01 20:35:00.126541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126553 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerName="util" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126559 4477 cpu_manager.go:414] "containerMap: removing container" podUID="12b6dd46-9985-44f9-a905-d904087774d1" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126563 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b6dd46-9985-44f9-a905-d904087774d1" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126569 4477 cpu_manager.go:414] "containerMap: removing container" podUID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126573 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126579 4477 cpu_manager.go:414] "containerMap: removing container" podUID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126583 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126588 4477 cpu_manager.go:414] "containerMap: removing container" podUID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126592 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126597 4477 cpu_manager.go:414] "containerMap: removing container" podUID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126601 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126608 4477 cpu_manager.go:414] "containerMap: removing container" podUID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126612 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126617 4477 cpu_manager.go:414] "containerMap: removing container" podUID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126621 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126628 4477 cpu_manager.go:414] "containerMap: removing container" podUID="12b6dd46-9985-44f9-a905-d904087774d1" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126631 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b6dd46-9985-44f9-a905-d904087774d1" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126642 4477 cpu_manager.go:414] "containerMap: removing container" podUID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerName="util" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126645 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerName="util" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126651 4477 cpu_manager.go:414] "containerMap: removing container" podUID="84010e72-32ff-487e-ad03-7c6608757314" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126654 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84010e72-32ff-487e-ad03-7c6608757314" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126661 4477 cpu_manager.go:414] "containerMap: removing container" podUID="12b6dd46-9985-44f9-a905-d904087774d1" containerName="util" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126665 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b6dd46-9985-44f9-a905-d904087774d1" containerName="util" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126670 4477 cpu_manager.go:414] "containerMap: removing container" podUID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerName="util" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126674 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerName="util" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126681 4477 cpu_manager.go:414] "containerMap: removing container" podUID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126685 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerName="pull" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126690 4477 cpu_manager.go:414] "containerMap: removing container" podUID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126693 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126698 4477 cpu_manager.go:414] "containerMap: removing container" podUID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126704 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126745 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="11317270-0e35-445c-8ffb-ca52c7ecc452" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126751 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="967c85c5-ef28-4ebc-bb49-62f0df3c23e9" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126757 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="84010e72-32ff-487e-ad03-7c6608757314" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126763 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b16c78d-103c-438e-ad5d-103fc6e68ac2" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126768 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4a0a6b-3415-4f6a-8bcb-fef9d9732f53" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126773 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8d31f0-4616-4e2f-b4dc-4a0af73bf1f3" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126779 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b6dd46-9985-44f9-a905-d904087774d1" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126785 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce8176c-9f10-41d0-91d4-1655bd2d59e5" containerName="extract" Nov 01 20:35:00.127381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.126790 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a978b7c-ca9f-4391-b26d-53ba29f18c2e" containerName="extract" Nov 01 20:35:00.148060 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.148042 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.205233 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.205201 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 20:35:00.205416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.205229 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 20:35:00.205416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.205262 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 20:35:00.205416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.205266 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 20:35:00.205416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.205304 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 20:35:00.205593 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.205443 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-46fcz\"" Nov 01 20:35:00.257158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.257125 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v"] Nov 01 20:35:00.331241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.331211 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32bfa74d-4d0d-4632-8bec-509e02e62f07-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.331436 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.331258 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.331436 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.331331 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.331436 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.331371 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32bfa74d-4d0d-4632-8bec-509e02e62f07-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.331557 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.331455 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.331557 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.331483 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7txg\" (UniqueName: \"kubernetes.io/projected/32bfa74d-4d0d-4632-8bec-509e02e62f07-kube-api-access-m7txg\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.432952 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.432871 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.432952 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.432905 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7txg\" (UniqueName: \"kubernetes.io/projected/32bfa74d-4d0d-4632-8bec-509e02e62f07-kube-api-access-m7txg\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.432952 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.432931 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32bfa74d-4d0d-4632-8bec-509e02e62f07-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.433263 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.432962 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.433263 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.432985 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.433263 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.433013 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32bfa74d-4d0d-4632-8bec-509e02e62f07-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.433416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.433376 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32bfa74d-4d0d-4632-8bec-509e02e62f07-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.435741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.435713 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32bfa74d-4d0d-4632-8bec-509e02e62f07-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.435879 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.435773 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.435879 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.435776 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.435879 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.435812 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32bfa74d-4d0d-4632-8bec-509e02e62f07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.504969 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.504932 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7txg\" (UniqueName: \"kubernetes.io/projected/32bfa74d-4d0d-4632-8bec-509e02e62f07-kube-api-access-m7txg\") pod \"opendatahub-operator-controller-manager-bd887889c-7sk4v\" (UID: \"32bfa74d-4d0d-4632-8bec-509e02e62f07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:00.797103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:00.796989 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:01.416934 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:01.416889 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bfa74d_4d0d_4632_8bec_509e02e62f07.slice/crio-e42bb8dd2e28f6402ec7786069809900abba61a09713c7e64329d74b81c14f8c WatchSource:0}: Error finding container e42bb8dd2e28f6402ec7786069809900abba61a09713c7e64329d74b81c14f8c: Status 404 returned error can't find the container with id e42bb8dd2e28f6402ec7786069809900abba61a09713c7e64329d74b81c14f8c Nov 01 20:35:01.463854 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:01.463814 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v"] Nov 01 20:35:01.929293 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:01.929260 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" event={"ID":"32bfa74d-4d0d-4632-8bec-509e02e62f07","Type":"ContainerStarted","Data":"e42bb8dd2e28f6402ec7786069809900abba61a09713c7e64329d74b81c14f8c"} Nov 01 20:35:06.295315 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.295271 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q"] Nov 01 20:35:06.297282 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.297258 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" Nov 01 20:35:06.380679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.380642 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1591e31d-1d7f-449d-acd8-01e6cebb949b-tmp\") pod \"jobset-operator-674cd59dc5-xbs7q\" (UID: \"1591e31d-1d7f-449d-acd8-01e6cebb949b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" Nov 01 20:35:06.380679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.380683 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w64mq\" (UniqueName: \"kubernetes.io/projected/1591e31d-1d7f-449d-acd8-01e6cebb949b-kube-api-access-w64mq\") pod \"jobset-operator-674cd59dc5-xbs7q\" (UID: \"1591e31d-1d7f-449d-acd8-01e6cebb949b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" Nov 01 20:35:06.399128 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.399048 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 20:35:06.399128 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.399095 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:35:06.399128 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.399111 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-bf56t\"" Nov 01 20:35:06.401047 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.401021 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q"] Nov 01 20:35:06.481990 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.481948 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1591e31d-1d7f-449d-acd8-01e6cebb949b-tmp\") pod \"jobset-operator-674cd59dc5-xbs7q\" (UID: \"1591e31d-1d7f-449d-acd8-01e6cebb949b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" Nov 01 20:35:06.482206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.481999 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w64mq\" (UniqueName: \"kubernetes.io/projected/1591e31d-1d7f-449d-acd8-01e6cebb949b-kube-api-access-w64mq\") pod \"jobset-operator-674cd59dc5-xbs7q\" (UID: \"1591e31d-1d7f-449d-acd8-01e6cebb949b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" Nov 01 20:35:06.482428 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.482399 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1591e31d-1d7f-449d-acd8-01e6cebb949b-tmp\") pod \"jobset-operator-674cd59dc5-xbs7q\" (UID: \"1591e31d-1d7f-449d-acd8-01e6cebb949b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" Nov 01 20:35:06.540058 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.540024 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w64mq\" (UniqueName: \"kubernetes.io/projected/1591e31d-1d7f-449d-acd8-01e6cebb949b-kube-api-access-w64mq\") pod \"jobset-operator-674cd59dc5-xbs7q\" (UID: \"1591e31d-1d7f-449d-acd8-01e6cebb949b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" Nov 01 20:35:06.559068 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:35:06.558984 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:35:06.625457 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.625424 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" Nov 01 20:35:06.953586 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:06.953555 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" event={"ID":"32bfa74d-4d0d-4632-8bec-509e02e62f07","Type":"ContainerStarted","Data":"cda31ebe809e7a07bd27e824b2e2b2e930de912d94d9e5a4e57b15d5ce587c49"} Nov 01 20:35:07.006163 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:07.006133 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q"] Nov 01 20:35:07.008637 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:07.008609 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1591e31d_1d7f_449d_acd8_01e6cebb949b.slice/crio-3114cb3f1c82434b85403c4c900096b199996fafde2c61b308ae8fa3f3c9ad33 WatchSource:0}: Error finding container 3114cb3f1c82434b85403c4c900096b199996fafde2c61b308ae8fa3f3c9ad33: Status 404 returned error can't find the container with id 3114cb3f1c82434b85403c4c900096b199996fafde2c61b308ae8fa3f3c9ad33 Nov 01 20:35:07.962576 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:07.962537 4477 generic.go:356] "Generic (PLEG): container finished" podID="32bfa74d-4d0d-4632-8bec-509e02e62f07" containerID="cda31ebe809e7a07bd27e824b2e2b2e930de912d94d9e5a4e57b15d5ce587c49" exitCode=0 Nov 01 20:35:07.963059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:07.962723 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" event={"ID":"32bfa74d-4d0d-4632-8bec-509e02e62f07","Type":"ContainerDied","Data":"cda31ebe809e7a07bd27e824b2e2b2e930de912d94d9e5a4e57b15d5ce587c49"} Nov 01 20:35:07.964835 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:07.964802 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" event={"ID":"1591e31d-1d7f-449d-acd8-01e6cebb949b","Type":"ContainerStarted","Data":"3114cb3f1c82434b85403c4c900096b199996fafde2c61b308ae8fa3f3c9ad33"} Nov 01 20:35:08.972430 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:08.972394 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" event={"ID":"32bfa74d-4d0d-4632-8bec-509e02e62f07","Type":"ContainerStarted","Data":"a16dc92454e8abd65c247819d81112dd19ebaa2f0478d7653ed4f2326fcd3207"} Nov 01 20:35:08.972883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:08.972543 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:09.209805 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:09.209756 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" podStartSLOduration=3.4531643819999998 podStartE2EDuration="10.209741102s" totalImagesPullingTime="6.268196493s" totalInitContainerRuntime="488.380227ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:09.158645834 +0000 UTC m=+262.808697391" watchObservedRunningTime="2026-11-01 20:35:09.209741102 +0000 UTC m=+262.859792646" Nov 01 20:35:10.315117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.315064 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft"] Nov 01 20:35:10.317100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.317067 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" Nov 01 20:35:10.371735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.371665 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:35:10.371735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.371665 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 20:35:10.371936 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.371748 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vvqzf\"" Nov 01 20:35:10.411853 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.411768 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhnk5\" (UniqueName: \"kubernetes.io/projected/58e64a52-eba7-4280-a4be-78b3d6c968b2-kube-api-access-dhnk5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft\" (UID: \"58e64a52-eba7-4280-a4be-78b3d6c968b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" Nov 01 20:35:10.411853 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.411827 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e64a52-eba7-4280-a4be-78b3d6c968b2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft\" (UID: \"58e64a52-eba7-4280-a4be-78b3d6c968b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" Nov 01 20:35:10.422774 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.422746 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft"] Nov 01 20:35:10.512891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.512859 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e64a52-eba7-4280-a4be-78b3d6c968b2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft\" (UID: \"58e64a52-eba7-4280-a4be-78b3d6c968b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" Nov 01 20:35:10.513064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.512930 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhnk5\" (UniqueName: \"kubernetes.io/projected/58e64a52-eba7-4280-a4be-78b3d6c968b2-kube-api-access-dhnk5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft\" (UID: \"58e64a52-eba7-4280-a4be-78b3d6c968b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" Nov 01 20:35:10.513284 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.513266 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e64a52-eba7-4280-a4be-78b3d6c968b2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft\" (UID: \"58e64a52-eba7-4280-a4be-78b3d6c968b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" Nov 01 20:35:10.574006 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.573969 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhnk5\" (UniqueName: \"kubernetes.io/projected/58e64a52-eba7-4280-a4be-78b3d6c968b2-kube-api-access-dhnk5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft\" (UID: \"58e64a52-eba7-4280-a4be-78b3d6c968b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" Nov 01 20:35:10.627180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.627142 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" Nov 01 20:35:10.983944 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:10.983860 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" event={"ID":"1591e31d-1d7f-449d-acd8-01e6cebb949b","Type":"ContainerStarted","Data":"fb5fff0accc70859cfbc3d7fc4aa0cb32e224766f444103a9a22a592ab131a23"} Nov 01 20:35:11.055746 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:11.055721 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft"] Nov 01 20:35:11.057914 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:11.057885 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e64a52_eba7_4280_a4be_78b3d6c968b2.slice/crio-600fb1c6f352a083063feda92c19642c9c2e062172de60d1f9f8df4e3a832f67 WatchSource:0}: Error finding container 600fb1c6f352a083063feda92c19642c9c2e062172de60d1f9f8df4e3a832f67: Status 404 returned error can't find the container with id 600fb1c6f352a083063feda92c19642c9c2e062172de60d1f9f8df4e3a832f67 Nov 01 20:35:11.104351 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:11.104295 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xbs7q" podStartSLOduration=1.964738455 podStartE2EDuration="5.104281562s" totalImagesPullingTime="3.139543107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:11.102564136 +0000 UTC m=+264.752615708" watchObservedRunningTime="2026-11-01 20:35:11.104281562 +0000 UTC m=+264.754333106" Nov 01 20:35:11.989988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:11.989948 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" event={"ID":"58e64a52-eba7-4280-a4be-78b3d6c968b2","Type":"ContainerStarted","Data":"600fb1c6f352a083063feda92c19642c9c2e062172de60d1f9f8df4e3a832f67"} Nov 01 20:35:15.007414 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:15.007378 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" event={"ID":"58e64a52-eba7-4280-a4be-78b3d6c968b2","Type":"ContainerStarted","Data":"401fccd4618eed45afe59f1477f9518a62717bffb0ebf388d0b82c0a8db25a4f"} Nov 01 20:35:15.226062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:15.226000 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hv5ft" podStartSLOduration=1.9942785669999998 podStartE2EDuration="5.225982045s" totalImagesPullingTime="3.231703478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:15.225687401 +0000 UTC m=+268.875738966" watchObservedRunningTime="2026-11-01 20:35:15.225982045 +0000 UTC m=+268.876033591" Nov 01 20:35:19.662067 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.662027 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7"] Nov 01 20:35:19.665398 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.665378 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" Nov 01 20:35:19.691010 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.690974 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbk9r\" (UniqueName: \"kubernetes.io/projected/ccc35dab-0d20-49dc-abee-3c696bedfed1-kube-api-access-gbk9r\") pod \"cert-manager-cainjector-5c57bbd4fc-lnmc7\" (UID: \"ccc35dab-0d20-49dc-abee-3c696bedfed1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" Nov 01 20:35:19.691183 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.691033 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccc35dab-0d20-49dc-abee-3c696bedfed1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lnmc7\" (UID: \"ccc35dab-0d20-49dc-abee-3c696bedfed1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" Nov 01 20:35:19.717513 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.717484 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 20:35:19.767294 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.767265 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 20:35:19.767294 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.767265 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cqfd9\"" Nov 01 20:35:19.770305 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.770282 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7"] Nov 01 20:35:19.792495 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.792464 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbk9r\" (UniqueName: \"kubernetes.io/projected/ccc35dab-0d20-49dc-abee-3c696bedfed1-kube-api-access-gbk9r\") pod \"cert-manager-cainjector-5c57bbd4fc-lnmc7\" (UID: \"ccc35dab-0d20-49dc-abee-3c696bedfed1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" Nov 01 20:35:19.792677 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.792543 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccc35dab-0d20-49dc-abee-3c696bedfed1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lnmc7\" (UID: \"ccc35dab-0d20-49dc-abee-3c696bedfed1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" Nov 01 20:35:19.855844 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.855813 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbk9r\" (UniqueName: \"kubernetes.io/projected/ccc35dab-0d20-49dc-abee-3c696bedfed1-kube-api-access-gbk9r\") pod \"cert-manager-cainjector-5c57bbd4fc-lnmc7\" (UID: \"ccc35dab-0d20-49dc-abee-3c696bedfed1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" Nov 01 20:35:19.856181 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.856162 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccc35dab-0d20-49dc-abee-3c696bedfed1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lnmc7\" (UID: \"ccc35dab-0d20-49dc-abee-3c696bedfed1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" Nov 01 20:35:19.911202 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.911168 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z"] Nov 01 20:35:19.914524 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.914458 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:19.968442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.968408 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 20:35:19.968635 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.968553 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sg6qc\"" Nov 01 20:35:19.976762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.976736 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" Nov 01 20:35:19.979196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.979174 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z"] Nov 01 20:35:19.981198 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.981178 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-7sk4v" Nov 01 20:35:19.994012 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.993982 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgkk\" (UniqueName: \"kubernetes.io/projected/94ee16ee-c2be-4914-885a-019c2bef3020-kube-api-access-qhgkk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-79v4z\" (UID: \"94ee16ee-c2be-4914-885a-019c2bef3020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:19.994130 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:19.994017 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee16ee-c2be-4914-885a-019c2bef3020-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-79v4z\" (UID: \"94ee16ee-c2be-4914-885a-019c2bef3020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:20.068570 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:20.068539 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 20:35:20.095110 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:20.095068 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgkk\" (UniqueName: \"kubernetes.io/projected/94ee16ee-c2be-4914-885a-019c2bef3020-kube-api-access-qhgkk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-79v4z\" (UID: \"94ee16ee-c2be-4914-885a-019c2bef3020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:20.095300 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:20.095118 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee16ee-c2be-4914-885a-019c2bef3020-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-79v4z\" (UID: \"94ee16ee-c2be-4914-885a-019c2bef3020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:20.095580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:20.095553 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee16ee-c2be-4914-885a-019c2bef3020-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-79v4z\" (UID: \"94ee16ee-c2be-4914-885a-019c2bef3020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:20.156757 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:20.156725 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgkk\" (UniqueName: \"kubernetes.io/projected/94ee16ee-c2be-4914-885a-019c2bef3020-kube-api-access-qhgkk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-79v4z\" (UID: \"94ee16ee-c2be-4914-885a-019c2bef3020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:20.227379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:20.227303 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:20.357635 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:20.357572 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7"] Nov 01 20:35:20.360104 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:20.360055 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc35dab_0d20_49dc_abee_3c696bedfed1.slice/crio-bdf1f517624207acb6a0d0038b51243e504d9dae7e0bebfdfdd376c47a5a0763 WatchSource:0}: Error finding container bdf1f517624207acb6a0d0038b51243e504d9dae7e0bebfdfdd376c47a5a0763: Status 404 returned error can't find the container with id bdf1f517624207acb6a0d0038b51243e504d9dae7e0bebfdfdd376c47a5a0763 Nov 01 20:35:20.612539 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:20.612517 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z"] Nov 01 20:35:20.614658 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:20.614628 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ee16ee_c2be_4914_885a_019c2bef3020.slice/crio-22e5b13613706f703c1de63e15a60b3938d4248016f46fc3587e5b4112ddfdd8 WatchSource:0}: Error finding container 22e5b13613706f703c1de63e15a60b3938d4248016f46fc3587e5b4112ddfdd8: Status 404 returned error can't find the container with id 22e5b13613706f703c1de63e15a60b3938d4248016f46fc3587e5b4112ddfdd8 Nov 01 20:35:21.049331 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:21.049245 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" event={"ID":"94ee16ee-c2be-4914-885a-019c2bef3020","Type":"ContainerStarted","Data":"22e5b13613706f703c1de63e15a60b3938d4248016f46fc3587e5b4112ddfdd8"} Nov 01 20:35:21.050536 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:21.050513 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" event={"ID":"ccc35dab-0d20-49dc-abee-3c696bedfed1","Type":"ContainerStarted","Data":"bdf1f517624207acb6a0d0038b51243e504d9dae7e0bebfdfdd376c47a5a0763"} Nov 01 20:35:24.331155 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:24.331125 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j"] Nov 01 20:35:24.334356 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:24.334339 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" Nov 01 20:35:24.386460 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:24.386433 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dw7rq\"" Nov 01 20:35:24.432625 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:24.432583 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrw9x\" (UniqueName: \"kubernetes.io/projected/d41f48c6-2ec9-47f8-b009-b6bc66d9288e-kube-api-access-qrw9x\") pod \"authorino-operator-7b7bb4bc77-kcx4j\" (UID: \"d41f48c6-2ec9-47f8-b009-b6bc66d9288e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" Nov 01 20:35:24.438106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:24.438064 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j"] Nov 01 20:35:24.533892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:24.533855 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrw9x\" (UniqueName: \"kubernetes.io/projected/d41f48c6-2ec9-47f8-b009-b6bc66d9288e-kube-api-access-qrw9x\") pod \"authorino-operator-7b7bb4bc77-kcx4j\" (UID: \"d41f48c6-2ec9-47f8-b009-b6bc66d9288e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" Nov 01 20:35:24.592731 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:24.592656 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrw9x\" (UniqueName: \"kubernetes.io/projected/d41f48c6-2ec9-47f8-b009-b6bc66d9288e-kube-api-access-qrw9x\") pod \"authorino-operator-7b7bb4bc77-kcx4j\" (UID: \"d41f48c6-2ec9-47f8-b009-b6bc66d9288e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" Nov 01 20:35:24.650359 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:24.650318 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" Nov 01 20:35:25.049229 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:25.049151 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j"] Nov 01 20:35:25.052101 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:25.052051 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41f48c6_2ec9_47f8_b009_b6bc66d9288e.slice/crio-d56e26f8e4a06796b5e83a3fe4d1fea14eaca36e81f517d0c0b2d98854c65060 WatchSource:0}: Error finding container d56e26f8e4a06796b5e83a3fe4d1fea14eaca36e81f517d0c0b2d98854c65060: Status 404 returned error can't find the container with id d56e26f8e4a06796b5e83a3fe4d1fea14eaca36e81f517d0c0b2d98854c65060 Nov 01 20:35:25.076510 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:25.076474 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" event={"ID":"d41f48c6-2ec9-47f8-b009-b6bc66d9288e","Type":"ContainerStarted","Data":"d56e26f8e4a06796b5e83a3fe4d1fea14eaca36e81f517d0c0b2d98854c65060"} Nov 01 20:35:27.091311 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.091254 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" event={"ID":"ccc35dab-0d20-49dc-abee-3c696bedfed1","Type":"ContainerStarted","Data":"85a22f4da32eb462d257ee3f314f81e0ddb992b5a49908bb529e4c5258c1185a"} Nov 01 20:35:27.206327 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.206262 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lnmc7" podStartSLOduration=2.545539746 podStartE2EDuration="8.206243364s" totalImagesPullingTime="5.660703618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:27.204955395 +0000 UTC m=+280.855006941" watchObservedRunningTime="2026-11-01 20:35:27.206243364 +0000 UTC m=+280.856294910" Nov 01 20:35:27.564602 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.564564 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc"] Nov 01 20:35:27.568568 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.568526 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" Nov 01 20:35:27.627798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.627488 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kwxx6\"" Nov 01 20:35:27.664366 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.664325 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fd95\" (UniqueName: \"kubernetes.io/projected/1e6e09bf-8759-4f40-931d-fcd5d73146c9-kube-api-access-9fd95\") pod \"dns-operator-controller-manager-8557cbdb84-255lc\" (UID: \"1e6e09bf-8759-4f40-931d-fcd5d73146c9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" Nov 01 20:35:27.679378 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.679352 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 20:35:27.681615 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.681578 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc"] Nov 01 20:35:27.765459 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.765412 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fd95\" (UniqueName: \"kubernetes.io/projected/1e6e09bf-8759-4f40-931d-fcd5d73146c9-kube-api-access-9fd95\") pod \"dns-operator-controller-manager-8557cbdb84-255lc\" (UID: \"1e6e09bf-8759-4f40-931d-fcd5d73146c9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" Nov 01 20:35:27.827040 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.826964 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fd95\" (UniqueName: \"kubernetes.io/projected/1e6e09bf-8759-4f40-931d-fcd5d73146c9-kube-api-access-9fd95\") pod \"dns-operator-controller-manager-8557cbdb84-255lc\" (UID: \"1e6e09bf-8759-4f40-931d-fcd5d73146c9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" Nov 01 20:35:27.884527 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:27.884019 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" Nov 01 20:35:28.283060 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:28.283040 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc"] Nov 01 20:35:28.286590 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:28.286559 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6e09bf_8759_4f40_931d_fcd5d73146c9.slice/crio-11a366926678c153edad91b832e114bd729543e824cb627653608d9e27c41e69 WatchSource:0}: Error finding container 11a366926678c153edad91b832e114bd729543e824cb627653608d9e27c41e69: Status 404 returned error can't find the container with id 11a366926678c153edad91b832e114bd729543e824cb627653608d9e27c41e69 Nov 01 20:35:29.105061 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:29.105021 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" event={"ID":"94ee16ee-c2be-4914-885a-019c2bef3020","Type":"ContainerStarted","Data":"d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184"} Nov 01 20:35:29.105283 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:29.105134 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:29.106790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:29.106760 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" event={"ID":"d41f48c6-2ec9-47f8-b009-b6bc66d9288e","Type":"ContainerStarted","Data":"949340a703c03697ea94f3b8928478d7a7aa4d5814536da5b659a9acb1531fd5"} Nov 01 20:35:29.106873 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:29.106810 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" Nov 01 20:35:29.108097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:29.108053 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" event={"ID":"1e6e09bf-8759-4f40-931d-fcd5d73146c9","Type":"ContainerStarted","Data":"11a366926678c153edad91b832e114bd729543e824cb627653608d9e27c41e69"} Nov 01 20:35:29.223623 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:29.223572 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" podStartSLOduration=2.129833996 podStartE2EDuration="10.223558711s" totalImagesPullingTime="8.093724715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:29.22186505 +0000 UTC m=+282.871916594" watchObservedRunningTime="2026-11-01 20:35:29.223558711 +0000 UTC m=+282.873610259" Nov 01 20:35:29.338115 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:29.338048 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" podStartSLOduration=1.913381329 podStartE2EDuration="5.338032676s" totalImagesPullingTime="3.424651347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:29.337009233 +0000 UTC m=+282.987060796" watchObservedRunningTime="2026-11-01 20:35:29.338032676 +0000 UTC m=+282.988084220" Nov 01 20:35:31.011725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.011695 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2"] Nov 01 20:35:31.016562 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.016533 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" Nov 01 20:35:31.097660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.097625 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qln\" (UniqueName: \"kubernetes.io/projected/4da15d47-468c-483b-8047-2de04909f3fb-kube-api-access-57qln\") pod \"limitador-operator-controller-manager-5778bb4449-4cbs2\" (UID: \"4da15d47-468c-483b-8047-2de04909f3fb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" Nov 01 20:35:31.125918 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.125886 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vs7br\"" Nov 01 20:35:31.127785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.127763 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2"] Nov 01 20:35:31.199098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.199048 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qln\" (UniqueName: \"kubernetes.io/projected/4da15d47-468c-483b-8047-2de04909f3fb-kube-api-access-57qln\") pod \"limitador-operator-controller-manager-5778bb4449-4cbs2\" (UID: \"4da15d47-468c-483b-8047-2de04909f3fb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" Nov 01 20:35:31.260557 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.260525 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qln\" (UniqueName: \"kubernetes.io/projected/4da15d47-468c-483b-8047-2de04909f3fb-kube-api-access-57qln\") pod \"limitador-operator-controller-manager-5778bb4449-4cbs2\" (UID: \"4da15d47-468c-483b-8047-2de04909f3fb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" Nov 01 20:35:31.331279 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.331252 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" Nov 01 20:35:31.703650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:31.703563 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2"] Nov 01 20:35:31.706921 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:31.706896 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da15d47_468c_483b_8047_2de04909f3fb.slice/crio-0f4ae1429cb0c4dcb27e7aecc11f5ae8ca2398129219bbdc26866faced9c79d3 WatchSource:0}: Error finding container 0f4ae1429cb0c4dcb27e7aecc11f5ae8ca2398129219bbdc26866faced9c79d3: Status 404 returned error can't find the container with id 0f4ae1429cb0c4dcb27e7aecc11f5ae8ca2398129219bbdc26866faced9c79d3 Nov 01 20:35:32.124810 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:32.124776 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" event={"ID":"4da15d47-468c-483b-8047-2de04909f3fb","Type":"ContainerStarted","Data":"0f4ae1429cb0c4dcb27e7aecc11f5ae8ca2398129219bbdc26866faced9c79d3"} Nov 01 20:35:32.986496 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:32.986455 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt"] Nov 01 20:35:32.991139 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:32.991115 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.043126 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.043100 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jfvqj\"" Nov 01 20:35:33.094751 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.094721 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt"] Nov 01 20:35:33.118761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.118728 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.118761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.118763 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.118980 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.118791 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.118980 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.118817 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.118980 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.118910 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.119094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.118982 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.119094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.119019 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrnqr\" (UniqueName: \"kubernetes.io/projected/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-kube-api-access-mrnqr\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.119167 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.119113 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.119167 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.119133 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.219920 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.219889 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.219920 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.219923 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.219943 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.219961 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.219978 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220005 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220030 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrnqr\" (UniqueName: \"kubernetes.io/projected/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-kube-api-access-mrnqr\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220109 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220137 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220426 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220760 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220584 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220760 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220625 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220760 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220664 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.220856 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.220793 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.222580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.222555 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.222973 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.222958 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.276843 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.276781 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.277248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.277227 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrnqr\" (UniqueName: \"kubernetes.io/projected/1bb50d1b-3ccf-4e41-8caf-3be2712e4c19-kube-api-access-mrnqr\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt\" (UID: \"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.304839 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.304813 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:33.731319 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.731292 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt"] Nov 01 20:35:33.734101 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:33.734048 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb50d1b_3ccf_4e41_8caf_3be2712e4c19.slice/crio-dfc49d254a521f2b64c14196fd849d0600d45ab6932f97c9dfc0cb66554bc10a WatchSource:0}: Error finding container dfc49d254a521f2b64c14196fd849d0600d45ab6932f97c9dfc0cb66554bc10a: Status 404 returned error can't find the container with id dfc49d254a521f2b64c14196fd849d0600d45ab6932f97c9dfc0cb66554bc10a Nov 01 20:35:33.736814 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.736778 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:35:33.736896 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.736854 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:35:33.736896 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:33.736885 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:35:34.137851 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:34.137818 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" event={"ID":"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19","Type":"ContainerStarted","Data":"276c0199877e74d277cc1421535b52a8e080fc940f25b4ccf362d7c03cd80371"} Nov 01 20:35:34.138052 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:34.137872 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" event={"ID":"1bb50d1b-3ccf-4e41-8caf-3be2712e4c19","Type":"ContainerStarted","Data":"dfc49d254a521f2b64c14196fd849d0600d45ab6932f97c9dfc0cb66554bc10a"} Nov 01 20:35:34.301276 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:34.301220 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" podStartSLOduration=2.301205831 podStartE2EDuration="2.301205831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:34.254475559 +0000 UTC m=+287.904527129" watchObservedRunningTime="2026-11-01 20:35:34.301205831 +0000 UTC m=+287.951257376" Nov 01 20:35:34.305625 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:34.305601 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:34.310656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:34.310622 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:35.144726 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:35.144688 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" event={"ID":"4da15d47-468c-483b-8047-2de04909f3fb","Type":"ContainerStarted","Data":"9b0385c262d8b01f7d70b782c733292dbb4a8fd9b5890b9752c5e6f8a93d5ef0"} Nov 01 20:35:35.144932 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:35.144755 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" Nov 01 20:35:35.145000 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:35.144937 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:35.145982 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:35.145963 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt" Nov 01 20:35:35.281481 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:35.281409 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" podStartSLOduration=2.482785003 podStartE2EDuration="5.281388895s" totalImagesPullingTime="2.798603892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:35.268241856 +0000 UTC m=+288.918293418" watchObservedRunningTime="2026-11-01 20:35:35.281388895 +0000 UTC m=+288.931440439" Nov 01 20:35:36.552356 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:35:36.552315 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:35:38.161843 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:38.161811 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" event={"ID":"1e6e09bf-8759-4f40-931d-fcd5d73146c9","Type":"ContainerStarted","Data":"92e3a256f7c3a97f5470e33f950f76f5ea8ab907678b83723d090f7af609aecf"} Nov 01 20:35:38.162254 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:38.161924 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" Nov 01 20:35:38.275532 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:38.275432 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" podStartSLOduration=1.603010034 podStartE2EDuration="11.275416024s" totalImagesPullingTime="9.67240599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:38.274248602 +0000 UTC m=+291.924300166" watchObservedRunningTime="2026-11-01 20:35:38.275416024 +0000 UTC m=+291.925467569" Nov 01 20:35:40.114971 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.114939 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kcx4j" Nov 01 20:35:40.115456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.115008 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:40.800186 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.800140 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g"] Nov 01 20:35:40.802636 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.802608 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:40.891489 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.891456 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716b0934-7ae5-439b-953c-f55dca1f8a5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bvn9g\" (UID: \"716b0934-7ae5-439b-953c-f55dca1f8a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:40.891668 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.891589 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v455n\" (UniqueName: \"kubernetes.io/projected/716b0934-7ae5-439b-953c-f55dca1f8a5e-kube-api-access-v455n\") pod \"kuadrant-operator-controller-manager-678d87766f-bvn9g\" (UID: \"716b0934-7ae5-439b-953c-f55dca1f8a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:40.904013 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.903986 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g"] Nov 01 20:35:40.992248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.992215 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v455n\" (UniqueName: \"kubernetes.io/projected/716b0934-7ae5-439b-953c-f55dca1f8a5e-kube-api-access-v455n\") pod \"kuadrant-operator-controller-manager-678d87766f-bvn9g\" (UID: \"716b0934-7ae5-439b-953c-f55dca1f8a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:40.992404 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.992274 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716b0934-7ae5-439b-953c-f55dca1f8a5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bvn9g\" (UID: \"716b0934-7ae5-439b-953c-f55dca1f8a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:40.992640 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:40.992621 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716b0934-7ae5-439b-953c-f55dca1f8a5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bvn9g\" (UID: \"716b0934-7ae5-439b-953c-f55dca1f8a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:41.051771 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:41.051688 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v455n\" (UniqueName: \"kubernetes.io/projected/716b0934-7ae5-439b-953c-f55dca1f8a5e-kube-api-access-v455n\") pod \"kuadrant-operator-controller-manager-678d87766f-bvn9g\" (UID: \"716b0934-7ae5-439b-953c-f55dca1f8a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:41.116609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:41.116574 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:41.549140 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:41.549113 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g"] Nov 01 20:35:41.551905 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:41.551878 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716b0934_7ae5_439b_953c_f55dca1f8a5e.slice/crio-e2435a86585ff035bf2e36b58a1dfa8cedcc447b5ad732513299c2f2beebaaca WatchSource:0}: Error finding container e2435a86585ff035bf2e36b58a1dfa8cedcc447b5ad732513299c2f2beebaaca: Status 404 returned error can't find the container with id e2435a86585ff035bf2e36b58a1dfa8cedcc447b5ad732513299c2f2beebaaca Nov 01 20:35:42.185262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:42.185215 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" event={"ID":"716b0934-7ae5-439b-953c-f55dca1f8a5e","Type":"ContainerStarted","Data":"3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445"} Nov 01 20:35:42.185262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:42.185263 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" event={"ID":"716b0934-7ae5-439b-953c-f55dca1f8a5e","Type":"ContainerStarted","Data":"e2435a86585ff035bf2e36b58a1dfa8cedcc447b5ad732513299c2f2beebaaca"} Nov 01 20:35:42.185701 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:42.185348 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:42.403909 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:42.403852 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" podStartSLOduration=2.403838132 podStartE2EDuration="2.403838132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:42.35306482 +0000 UTC m=+296.003116361" watchObservedRunningTime="2026-11-01 20:35:42.403838132 +0000 UTC m=+296.053889676" Nov 01 20:35:46.152899 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:46.152866 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4cbs2" Nov 01 20:35:49.169998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:49.169966 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-255lc" Nov 01 20:35:53.192937 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.192903 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 20:35:53.415669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.415619 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z"] Nov 01 20:35:53.415904 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.415869 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" podUID="94ee16ee-c2be-4914-885a-019c2bef3020" containerName="manager" containerID="cri-o://d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184" gracePeriod=10 Nov 01 20:35:53.768777 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.768746 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:53.909690 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.909601 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ee16ee-c2be-4914-885a-019c2bef3020-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee16ee-c2be-4914-885a-019c2bef3020-extensions-socket-volume\") pod \"94ee16ee-c2be-4914-885a-019c2bef3020\" (UID: \"94ee16ee-c2be-4914-885a-019c2bef3020\") " Nov 01 20:35:53.909874 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.909717 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ee16ee-c2be-4914-885a-019c2bef3020-kube-api-access-qhgkk\" (UniqueName: \"kubernetes.io/projected/94ee16ee-c2be-4914-885a-019c2bef3020-kube-api-access-qhgkk\") pod \"94ee16ee-c2be-4914-885a-019c2bef3020\" (UID: \"94ee16ee-c2be-4914-885a-019c2bef3020\") " Nov 01 20:35:53.909954 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.909929 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ee16ee-c2be-4914-885a-019c2bef3020-extensions-socket-volume" pod "94ee16ee-c2be-4914-885a-019c2bef3020" (UID: "94ee16ee-c2be-4914-885a-019c2bef3020"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:35:53.910035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.910021 4477 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee16ee-c2be-4914-885a-019c2bef3020-extensions-socket-volume\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:53.912221 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:53.912191 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ee16ee-c2be-4914-885a-019c2bef3020-kube-api-access-qhgkk" pod "94ee16ee-c2be-4914-885a-019c2bef3020" (UID: "94ee16ee-c2be-4914-885a-019c2bef3020"). InnerVolumeSpecName "kube-api-access-qhgkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:35:54.010938 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.010905 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgkk\" (UniqueName: \"kubernetes.io/projected/94ee16ee-c2be-4914-885a-019c2bef3020-kube-api-access-qhgkk\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:54.243360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.243270 4477 generic.go:356] "Generic (PLEG): container finished" podID="94ee16ee-c2be-4914-885a-019c2bef3020" containerID="d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184" exitCode=0 Nov 01 20:35:54.243360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.243340 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" Nov 01 20:35:54.243829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.243363 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" event={"ID":"94ee16ee-c2be-4914-885a-019c2bef3020","Type":"ContainerDied","Data":"d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184"} Nov 01 20:35:54.243829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.243441 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z" event={"ID":"94ee16ee-c2be-4914-885a-019c2bef3020","Type":"ContainerDied","Data":"22e5b13613706f703c1de63e15a60b3938d4248016f46fc3587e5b4112ddfdd8"} Nov 01 20:35:54.243829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.243464 4477 scope.go:122] "RemoveContainer" containerID="d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184" Nov 01 20:35:54.252526 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.252505 4477 scope.go:122] "RemoveContainer" containerID="d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184" Nov 01 20:35:54.252811 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:35:54.252789 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184\": container with ID starting with d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184 not found: ID does not exist" containerID="d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184" Nov 01 20:35:54.252855 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.252822 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184"} err="failed to get container status \"d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184\": rpc error: code = NotFound desc = could not find container \"d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184\": container with ID starting with d0abf775e61efb0be710007d23ea469b6dece161eb343f735917dd8427618184 not found: ID does not exist" Nov 01 20:35:54.410775 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.410742 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z"] Nov 01 20:35:54.460569 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.460541 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-79v4z"] Nov 01 20:35:54.503034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:54.502970 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ee16ee-c2be-4914-885a-019c2bef3020" path="/var/lib/kubelet/pods/94ee16ee-c2be-4914-885a-019c2bef3020/volumes" Nov 01 20:35:56.001001 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.000968 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9krhc"] Nov 01 20:35:56.001419 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.001355 4477 cpu_manager.go:414] "containerMap: removing container" podUID="94ee16ee-c2be-4914-885a-019c2bef3020" containerName="manager" Nov 01 20:35:56.001419 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.001367 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ee16ee-c2be-4914-885a-019c2bef3020" containerName="manager" Nov 01 20:35:56.001501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.001448 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ee16ee-c2be-4914-885a-019c2bef3020" containerName="manager" Nov 01 20:35:56.005832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.005806 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9krhc" Nov 01 20:35:56.058927 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.058895 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9krhc"] Nov 01 20:35:56.105613 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.105583 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmsnc\"" Nov 01 20:35:56.128840 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.128810 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596vx\" (UniqueName: \"kubernetes.io/projected/9b74b268-4742-4aba-a168-5c386de5f622-kube-api-access-596vx\") pod \"authorino-57f7f8659f-9krhc\" (UID: \"9b74b268-4742-4aba-a168-5c386de5f622\") " pod="kuadrant-system/authorino-57f7f8659f-9krhc" Nov 01 20:35:56.229271 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.229236 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596vx\" (UniqueName: \"kubernetes.io/projected/9b74b268-4742-4aba-a168-5c386de5f622-kube-api-access-596vx\") pod \"authorino-57f7f8659f-9krhc\" (UID: \"9b74b268-4742-4aba-a168-5c386de5f622\") " pod="kuadrant-system/authorino-57f7f8659f-9krhc" Nov 01 20:35:56.286778 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.286701 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596vx\" (UniqueName: \"kubernetes.io/projected/9b74b268-4742-4aba-a168-5c386de5f622-kube-api-access-596vx\") pod \"authorino-57f7f8659f-9krhc\" (UID: \"9b74b268-4742-4aba-a168-5c386de5f622\") " pod="kuadrant-system/authorino-57f7f8659f-9krhc" Nov 01 20:35:56.317503 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.317475 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9krhc" Nov 01 20:35:56.695573 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.695549 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9krhc"] Nov 01 20:35:56.698724 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:35:56.698695 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b74b268_4742_4aba_a168_5c386de5f622.slice/crio-5d73acd4902cc041239ca5f45d133d12c4e1b19d3c2e8908309fc92567b0e527 WatchSource:0}: Error finding container 5d73acd4902cc041239ca5f45d133d12c4e1b19d3c2e8908309fc92567b0e527: Status 404 returned error can't find the container with id 5d73acd4902cc041239ca5f45d133d12c4e1b19d3c2e8908309fc92567b0e527 Nov 01 20:35:56.700067 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:56.700034 4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:35:57.268466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:35:57.268429 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9krhc" event={"ID":"9b74b268-4742-4aba-a168-5c386de5f622","Type":"ContainerStarted","Data":"5d73acd4902cc041239ca5f45d133d12c4e1b19d3c2e8908309fc92567b0e527"} Nov 01 20:36:00.287155 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:00.287119 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9krhc" event={"ID":"9b74b268-4742-4aba-a168-5c386de5f622","Type":"ContainerStarted","Data":"5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40"} Nov 01 20:36:00.399274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:00.399222 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-9krhc" podStartSLOduration=2.6776295169999997 podStartE2EDuration="5.399202634s" totalImagesPullingTime="2.721573117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:00.397191546 +0000 UTC m=+314.047243113" watchObservedRunningTime="2026-11-01 20:36:00.399202634 +0000 UTC m=+314.049254178" Nov 01 20:36:06.551502 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:36:06.551458 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:36:12.155562 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.155525 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:36:12.159448 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.159428 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:12.221900 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.221868 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 20:36:12.250131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.250094 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqpx\" (UniqueName: \"kubernetes.io/projected/2d9aff2d-e468-4399-ba75-ee34150146ba-kube-api-access-rjqpx\") pod \"limitador-limitador-7c9b87d574-k2jc8\" (UID: \"2d9aff2d-e468-4399-ba75-ee34150146ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:12.250312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.250147 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d9aff2d-e468-4399-ba75-ee34150146ba-config-file\") pod \"limitador-limitador-7c9b87d574-k2jc8\" (UID: \"2d9aff2d-e468-4399-ba75-ee34150146ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:12.271773 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.271742 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mb6k\"" Nov 01 20:36:12.274013 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.273988 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:36:12.350997 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.350962 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqpx\" (UniqueName: \"kubernetes.io/projected/2d9aff2d-e468-4399-ba75-ee34150146ba-kube-api-access-rjqpx\") pod \"limitador-limitador-7c9b87d574-k2jc8\" (UID: \"2d9aff2d-e468-4399-ba75-ee34150146ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:12.351207 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.351020 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d9aff2d-e468-4399-ba75-ee34150146ba-config-file\") pod \"limitador-limitador-7c9b87d574-k2jc8\" (UID: \"2d9aff2d-e468-4399-ba75-ee34150146ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:12.351685 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.351666 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d9aff2d-e468-4399-ba75-ee34150146ba-config-file\") pod \"limitador-limitador-7c9b87d574-k2jc8\" (UID: \"2d9aff2d-e468-4399-ba75-ee34150146ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:12.408046 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.407978 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqpx\" (UniqueName: \"kubernetes.io/projected/2d9aff2d-e468-4399-ba75-ee34150146ba-kube-api-access-rjqpx\") pod \"limitador-limitador-7c9b87d574-k2jc8\" (UID: \"2d9aff2d-e468-4399-ba75-ee34150146ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:12.461254 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.461217 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:36:12.470885 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.470865 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:12.848560 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:12.848497 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:36:12.851556 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:36:12.851524 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9aff2d_e468_4399_ba75_ee34150146ba.slice/crio-3c7c883febf3a2faa13b9f67631a20e05146b384a320c741a116871e44d3270e WatchSource:0}: Error finding container 3c7c883febf3a2faa13b9f67631a20e05146b384a320c741a116871e44d3270e: Status 404 returned error can't find the container with id 3c7c883febf3a2faa13b9f67631a20e05146b384a320c741a116871e44d3270e Nov 01 20:36:13.356989 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:13.356956 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" event={"ID":"2d9aff2d-e468-4399-ba75-ee34150146ba","Type":"ContainerStarted","Data":"3c7c883febf3a2faa13b9f67631a20e05146b384a320c741a116871e44d3270e"} Nov 01 20:36:15.891541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:15.891502 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9krhc"] Nov 01 20:36:15.891947 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:15.891775 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-9krhc" podUID="9b74b268-4742-4aba-a168-5c386de5f622" containerName="authorino" containerID="cri-o://5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40" gracePeriod=30 Nov 01 20:36:16.126237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.126209 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9krhc" Nov 01 20:36:16.184162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.183933 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b74b268-4742-4aba-a168-5c386de5f622-kube-api-access-596vx\" (UniqueName: \"kubernetes.io/projected/9b74b268-4742-4aba-a168-5c386de5f622-kube-api-access-596vx\") pod \"9b74b268-4742-4aba-a168-5c386de5f622\" (UID: \"9b74b268-4742-4aba-a168-5c386de5f622\") " Nov 01 20:36:16.190672 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.190631 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b74b268-4742-4aba-a168-5c386de5f622-kube-api-access-596vx" pod "9b74b268-4742-4aba-a168-5c386de5f622" (UID: "9b74b268-4742-4aba-a168-5c386de5f622"). InnerVolumeSpecName "kube-api-access-596vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:36:16.285450 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.285413 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596vx\" (UniqueName: \"kubernetes.io/projected/9b74b268-4742-4aba-a168-5c386de5f622-kube-api-access-596vx\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:16.374172 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.374131 4477 generic.go:356] "Generic (PLEG): container finished" podID="9b74b268-4742-4aba-a168-5c386de5f622" containerID="5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40" exitCode=0 Nov 01 20:36:16.374352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.374187 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9krhc" Nov 01 20:36:16.374352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.374215 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9krhc" event={"ID":"9b74b268-4742-4aba-a168-5c386de5f622","Type":"ContainerDied","Data":"5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40"} Nov 01 20:36:16.374352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.374261 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9krhc" event={"ID":"9b74b268-4742-4aba-a168-5c386de5f622","Type":"ContainerDied","Data":"5d73acd4902cc041239ca5f45d133d12c4e1b19d3c2e8908309fc92567b0e527"} Nov 01 20:36:16.374352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.374278 4477 scope.go:122] "RemoveContainer" containerID="5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40" Nov 01 20:36:16.383281 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.383262 4477 scope.go:122] "RemoveContainer" containerID="5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40" Nov 01 20:36:16.383555 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:36:16.383535 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40\": container with ID starting with 5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40 not found: ID does not exist" containerID="5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40" Nov 01 20:36:16.383600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.383565 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40"} err="failed to get container status \"5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40\": rpc error: code = NotFound desc = could not find container \"5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40\": container with ID starting with 5aeda6eae5cce5fc31259667373af0868d54295f2cbbb4e75f9ff1295bdd8a40 not found: ID does not exist" Nov 01 20:36:16.535112 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.535010 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9krhc"] Nov 01 20:36:16.539399 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:16.539373 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9krhc"] Nov 01 20:36:18.501874 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:18.501842 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b74b268-4742-4aba-a168-5c386de5f622" path="/var/lib/kubelet/pods/9b74b268-4742-4aba-a168-5c386de5f622/volumes" Nov 01 20:36:26.430657 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:26.430601 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" event={"ID":"2d9aff2d-e468-4399-ba75-ee34150146ba","Type":"ContainerStarted","Data":"a1b0b9ce084c6a7700848e143d5bcf2b29c7fdbc62bf967d6f7cf4999d484cd1"} Nov 01 20:36:26.431559 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:26.431528 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:26.545323 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:26.545273 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" podStartSLOduration=2.035662767 podStartE2EDuration="14.545259524s" totalImagesPullingTime="12.509596757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:26.543398913 +0000 UTC m=+340.193450452" watchObservedRunningTime="2026-11-01 20:36:26.545259524 +0000 UTC m=+340.195311069" Nov 01 20:36:36.552590 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:36:36.552549 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:36:38.443106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:38.443054 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-k2jc8" Nov 01 20:36:44.598440 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.598398 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp"] Nov 01 20:36:44.598843 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.598790 4477 cpu_manager.go:414] "containerMap: removing container" podUID="9b74b268-4742-4aba-a168-5c386de5f622" containerName="authorino" Nov 01 20:36:44.598843 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.598801 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b74b268-4742-4aba-a168-5c386de5f622" containerName="authorino" Nov 01 20:36:44.598915 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.598882 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b74b268-4742-4aba-a168-5c386de5f622" containerName="authorino" Nov 01 20:36:44.602713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.602695 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:44.656312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.656280 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:36:44.659165 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.659135 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp"] Nov 01 20:36:44.704966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.704930 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 20:36:44.705167 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.704974 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:36:44.705167 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.705007 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-48l2q\"" Nov 01 20:36:44.738700 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.738671 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpdk\" (UniqueName: \"kubernetes.io/projected/310295d3-536c-484b-acb6-757d91892c4a-kube-api-access-nvpdk\") pod \"postgres-7cdbf6dc44-bbjnp\" (UID: \"310295d3-536c-484b-acb6-757d91892c4a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:44.738877 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.738717 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/310295d3-536c-484b-acb6-757d91892c4a-data\") pod \"postgres-7cdbf6dc44-bbjnp\" (UID: \"310295d3-536c-484b-acb6-757d91892c4a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:44.839395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.839362 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpdk\" (UniqueName: \"kubernetes.io/projected/310295d3-536c-484b-acb6-757d91892c4a-kube-api-access-nvpdk\") pod \"postgres-7cdbf6dc44-bbjnp\" (UID: \"310295d3-536c-484b-acb6-757d91892c4a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:44.839572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.839410 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/310295d3-536c-484b-acb6-757d91892c4a-data\") pod \"postgres-7cdbf6dc44-bbjnp\" (UID: \"310295d3-536c-484b-acb6-757d91892c4a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:44.839795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.839780 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/310295d3-536c-484b-acb6-757d91892c4a-data\") pod \"postgres-7cdbf6dc44-bbjnp\" (UID: \"310295d3-536c-484b-acb6-757d91892c4a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:44.903490 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.903420 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpdk\" (UniqueName: \"kubernetes.io/projected/310295d3-536c-484b-acb6-757d91892c4a-kube-api-access-nvpdk\") pod \"postgres-7cdbf6dc44-bbjnp\" (UID: \"310295d3-536c-484b-acb6-757d91892c4a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:44.916143 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:44.916123 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:45.289595 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:45.289567 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp"] Nov 01 20:36:45.291983 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:36:45.291956 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310295d3_536c_484b_acb6_757d91892c4a.slice/crio-b36d1fca861628bae7bf499d9c63098a5d4e775d2f8c497f67b2977008c31aa9 WatchSource:0}: Error finding container b36d1fca861628bae7bf499d9c63098a5d4e775d2f8c497f67b2977008c31aa9: Status 404 returned error can't find the container with id b36d1fca861628bae7bf499d9c63098a5d4e775d2f8c497f67b2977008c31aa9 Nov 01 20:36:45.531599 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:45.531564 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" event={"ID":"310295d3-536c-484b-acb6-757d91892c4a","Type":"ContainerStarted","Data":"b36d1fca861628bae7bf499d9c63098a5d4e775d2f8c497f67b2977008c31aa9"} Nov 01 20:36:49.716442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.716405 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g52pk"] Nov 01 20:36:49.729594 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.729568 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:49.775330 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.775298 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g52pk"] Nov 01 20:36:49.822426 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.822399 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-w9fqt\"" Nov 01 20:36:49.822426 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.822399 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 20:36:49.830901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.830877 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 20:36:49.831180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.831156 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 20:36:49.833765 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.833739 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4"] Nov 01 20:36:49.839317 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.839294 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:49.883312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.883278 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa08a527-ddee-4c0f-866b-8e5d6ce4dfef-cert\") pod \"kserve-controller-manager-776cc66d47-g52pk\" (UID: \"fa08a527-ddee-4c0f-866b-8e5d6ce4dfef\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:49.883481 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.883319 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4lw\" (UniqueName: \"kubernetes.io/projected/fa08a527-ddee-4c0f-866b-8e5d6ce4dfef-kube-api-access-wv4lw\") pod \"kserve-controller-manager-776cc66d47-g52pk\" (UID: \"fa08a527-ddee-4c0f-866b-8e5d6ce4dfef\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:49.887286 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.887259 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4"] Nov 01 20:36:49.891582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.891563 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 20:36:49.891713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.891675 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-z2vqz\"" Nov 01 20:36:49.934457 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.934425 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hkzfs"] Nov 01 20:36:49.939384 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.939360 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:49.984262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.984192 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvwq\" (UniqueName: \"kubernetes.io/projected/32462b71-2673-4121-902f-a1167c23eaa9-kube-api-access-ldvwq\") pod \"llmisvc-controller-manager-869d5dcc5d-cnnc4\" (UID: \"32462b71-2673-4121-902f-a1167c23eaa9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:49.984262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.984230 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa08a527-ddee-4c0f-866b-8e5d6ce4dfef-cert\") pod \"kserve-controller-manager-776cc66d47-g52pk\" (UID: \"fa08a527-ddee-4c0f-866b-8e5d6ce4dfef\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:49.984262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.984254 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4lw\" (UniqueName: \"kubernetes.io/projected/fa08a527-ddee-4c0f-866b-8e5d6ce4dfef-kube-api-access-wv4lw\") pod \"kserve-controller-manager-776cc66d47-g52pk\" (UID: \"fa08a527-ddee-4c0f-866b-8e5d6ce4dfef\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:49.984536 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.984277 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32462b71-2673-4121-902f-a1167c23eaa9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-cnnc4\" (UID: \"32462b71-2673-4121-902f-a1167c23eaa9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:49.987169 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.987146 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa08a527-ddee-4c0f-866b-8e5d6ce4dfef-cert\") pod \"kserve-controller-manager-776cc66d47-g52pk\" (UID: \"fa08a527-ddee-4c0f-866b-8e5d6ce4dfef\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:49.995262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.995233 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 20:36:49.995436 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.995412 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-nt9hl\"" Nov 01 20:36:49.996015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:49.995997 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hkzfs"] Nov 01 20:36:50.043267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.043237 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4lw\" (UniqueName: \"kubernetes.io/projected/fa08a527-ddee-4c0f-866b-8e5d6ce4dfef-kube-api-access-wv4lw\") pod \"kserve-controller-manager-776cc66d47-g52pk\" (UID: \"fa08a527-ddee-4c0f-866b-8e5d6ce4dfef\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:50.046037 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.046012 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:50.084805 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.084769 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583e8cf3-8a77-46f3-85d8-40f64d4db713-tls-certs\") pod \"model-serving-api-75bd89fdfc-hkzfs\" (UID: \"583e8cf3-8a77-46f3-85d8-40f64d4db713\") " pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:50.084993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.084903 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvwq\" (UniqueName: \"kubernetes.io/projected/32462b71-2673-4121-902f-a1167c23eaa9-kube-api-access-ldvwq\") pod \"llmisvc-controller-manager-869d5dcc5d-cnnc4\" (UID: \"32462b71-2673-4121-902f-a1167c23eaa9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:50.084993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.084952 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwcl\" (UniqueName: \"kubernetes.io/projected/583e8cf3-8a77-46f3-85d8-40f64d4db713-kube-api-access-mjwcl\") pod \"model-serving-api-75bd89fdfc-hkzfs\" (UID: \"583e8cf3-8a77-46f3-85d8-40f64d4db713\") " pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:50.084993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.084975 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32462b71-2673-4121-902f-a1167c23eaa9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-cnnc4\" (UID: \"32462b71-2673-4121-902f-a1167c23eaa9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:50.087696 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.087671 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32462b71-2673-4121-902f-a1167c23eaa9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-cnnc4\" (UID: \"32462b71-2673-4121-902f-a1167c23eaa9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:50.142349 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.142323 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvwq\" (UniqueName: \"kubernetes.io/projected/32462b71-2673-4121-902f-a1167c23eaa9-kube-api-access-ldvwq\") pod \"llmisvc-controller-manager-869d5dcc5d-cnnc4\" (UID: \"32462b71-2673-4121-902f-a1167c23eaa9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:50.153872 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.153840 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:50.185694 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.185657 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwcl\" (UniqueName: \"kubernetes.io/projected/583e8cf3-8a77-46f3-85d8-40f64d4db713-kube-api-access-mjwcl\") pod \"model-serving-api-75bd89fdfc-hkzfs\" (UID: \"583e8cf3-8a77-46f3-85d8-40f64d4db713\") " pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:50.185883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.185755 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583e8cf3-8a77-46f3-85d8-40f64d4db713-tls-certs\") pod \"model-serving-api-75bd89fdfc-hkzfs\" (UID: \"583e8cf3-8a77-46f3-85d8-40f64d4db713\") " pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:50.188924 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.188888 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583e8cf3-8a77-46f3-85d8-40f64d4db713-tls-certs\") pod \"model-serving-api-75bd89fdfc-hkzfs\" (UID: \"583e8cf3-8a77-46f3-85d8-40f64d4db713\") " pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:50.243329 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.243249 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwcl\" (UniqueName: \"kubernetes.io/projected/583e8cf3-8a77-46f3-85d8-40f64d4db713-kube-api-access-mjwcl\") pod \"model-serving-api-75bd89fdfc-hkzfs\" (UID: \"583e8cf3-8a77-46f3-85d8-40f64d4db713\") " pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:50.254710 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.254676 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:50.426691 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.426656 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g52pk"] Nov 01 20:36:50.429662 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:36:50.429632 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa08a527_ddee_4c0f_866b_8e5d6ce4dfef.slice/crio-74a782c6f720b728539a63cfbd045b48d68de7b81f7e3b6d4fc4156caaa47117 WatchSource:0}: Error finding container 74a782c6f720b728539a63cfbd045b48d68de7b81f7e3b6d4fc4156caaa47117: Status 404 returned error can't find the container with id 74a782c6f720b728539a63cfbd045b48d68de7b81f7e3b6d4fc4156caaa47117 Nov 01 20:36:50.534688 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.534663 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4"] Nov 01 20:36:50.536419 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:36:50.536377 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32462b71_2673_4121_902f_a1167c23eaa9.slice/crio-658f9e5a5daaad2301f9f061860f4c5ceca5e4f5c70a68291d0214c6f5d898bd WatchSource:0}: Error finding container 658f9e5a5daaad2301f9f061860f4c5ceca5e4f5c70a68291d0214c6f5d898bd: Status 404 returned error can't find the container with id 658f9e5a5daaad2301f9f061860f4c5ceca5e4f5c70a68291d0214c6f5d898bd Nov 01 20:36:50.561710 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.561672 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" event={"ID":"32462b71-2673-4121-902f-a1167c23eaa9","Type":"ContainerStarted","Data":"658f9e5a5daaad2301f9f061860f4c5ceca5e4f5c70a68291d0214c6f5d898bd"} Nov 01 20:36:50.563238 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.563209 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" event={"ID":"fa08a527-ddee-4c0f-866b-8e5d6ce4dfef","Type":"ContainerStarted","Data":"74a782c6f720b728539a63cfbd045b48d68de7b81f7e3b6d4fc4156caaa47117"} Nov 01 20:36:50.639246 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:50.639165 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hkzfs"] Nov 01 20:36:50.642918 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:36:50.642890 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583e8cf3_8a77_46f3_85d8_40f64d4db713.slice/crio-85eff1fbbec980107bb3237401b96275b7433b375c727d342e9cc9a4fe45ad20 WatchSource:0}: Error finding container 85eff1fbbec980107bb3237401b96275b7433b375c727d342e9cc9a4fe45ad20: Status 404 returned error can't find the container with id 85eff1fbbec980107bb3237401b96275b7433b375c727d342e9cc9a4fe45ad20 Nov 01 20:36:51.574717 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:51.574679 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" event={"ID":"583e8cf3-8a77-46f3-85d8-40f64d4db713","Type":"ContainerStarted","Data":"85eff1fbbec980107bb3237401b96275b7433b375c727d342e9cc9a4fe45ad20"} Nov 01 20:36:53.535911 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:53.535872 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 20:36:54.602690 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:54.602642 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" event={"ID":"310295d3-536c-484b-acb6-757d91892c4a","Type":"ContainerStarted","Data":"59646c602f74528bbd702d2cedd9e9f126af7ab17d75f907ecae361e3d5b14fc"} Nov 01 20:36:54.603168 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:54.602827 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:36:54.604343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:54.604316 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" event={"ID":"583e8cf3-8a77-46f3-85d8-40f64d4db713","Type":"ContainerStarted","Data":"0378cd80e76676f1afbd91f9ee2b3a6a0514c805f2c6e0d50e27dab50e3618fe"} Nov 01 20:36:54.604462 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:54.604455 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:36:54.718179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:54.718130 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" podStartSLOduration=2.527332842 podStartE2EDuration="10.718115174s" totalImagesPullingTime="8.190782332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:54.715974159 +0000 UTC m=+368.366025704" watchObservedRunningTime="2026-11-01 20:36:54.718115174 +0000 UTC m=+368.368167064" Nov 01 20:36:54.833893 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:54.833831 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" podStartSLOduration=2.711633902 podStartE2EDuration="5.833814442s" totalImagesPullingTime="3.12218054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:54.831026779 +0000 UTC m=+368.481078327" watchObservedRunningTime="2026-11-01 20:36:54.833814442 +0000 UTC m=+368.483865987" Nov 01 20:36:55.610783 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:55.610744 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" event={"ID":"fa08a527-ddee-4c0f-866b-8e5d6ce4dfef","Type":"ContainerStarted","Data":"ebf7a8465fb708b63ad67a358a59b627465755263e2dace044a68dc150ec1d7a"} Nov 01 20:36:55.724591 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:55.724536 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" podStartSLOduration=2.459975896 podStartE2EDuration="6.724520135s" totalImagesPullingTime="4.264544239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:55.72339153 +0000 UTC m=+369.373443104" watchObservedRunningTime="2026-11-01 20:36:55.724520135 +0000 UTC m=+369.374571680" Nov 01 20:36:56.617356 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:56.617320 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:36:57.623274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:57.623229 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" event={"ID":"32462b71-2673-4121-902f-a1167c23eaa9","Type":"ContainerStarted","Data":"db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5"} Nov 01 20:36:57.623927 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:57.623902 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:36:57.739993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:36:57.739935 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" podStartSLOduration=2.5130777330000003 podStartE2EDuration="8.739917676s" totalImagesPullingTime="6.226839943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:57.737839909 +0000 UTC m=+371.387891476" watchObservedRunningTime="2026-11-01 20:36:57.739917676 +0000 UTC m=+371.389969221" Nov 01 20:37:00.641467 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:37:00.641441 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bbjnp" Nov 01 20:37:05.615190 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:37:05.615163 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-hkzfs" Nov 01 20:37:06.552291 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:37:06.552253 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:37:27.628442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:37:27.628364 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-g52pk" Nov 01 20:37:28.631033 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:37:28.630995 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:37:36.551658 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:37:36.551620 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:37:55.677887 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:37:55.677849 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:38:02.658042 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.658007 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qr74p"] Nov 01 20:38:02.660409 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.660393 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" Nov 01 20:38:02.713060 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.713017 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw727\" (UniqueName: \"kubernetes.io/projected/a6a1e955-1da5-420c-86ac-99dc980bda42-kube-api-access-vw727\") pod \"authorino-6f6944bcdd-qr74p\" (UID: \"a6a1e955-1da5-420c-86ac-99dc980bda42\") " pod="kuadrant-system/authorino-6f6944bcdd-qr74p" Nov 01 20:38:02.716270 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.716245 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qr74p"] Nov 01 20:38:02.762956 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.762925 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmsnc\"" Nov 01 20:38:02.813612 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.813580 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw727\" (UniqueName: \"kubernetes.io/projected/a6a1e955-1da5-420c-86ac-99dc980bda42-kube-api-access-vw727\") pod \"authorino-6f6944bcdd-qr74p\" (UID: \"a6a1e955-1da5-420c-86ac-99dc980bda42\") " pod="kuadrant-system/authorino-6f6944bcdd-qr74p" Nov 01 20:38:02.870546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.870507 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw727\" (UniqueName: \"kubernetes.io/projected/a6a1e955-1da5-420c-86ac-99dc980bda42-kube-api-access-vw727\") pod \"authorino-6f6944bcdd-qr74p\" (UID: \"a6a1e955-1da5-420c-86ac-99dc980bda42\") " pod="kuadrant-system/authorino-6f6944bcdd-qr74p" Nov 01 20:38:02.971627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.971552 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" Nov 01 20:38:02.977280 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:02.977253 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qr74p"] Nov 01 20:38:03.000481 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.000449 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f66647665-4cn7k"] Nov 01 20:38:03.003547 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.003526 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f66647665-4cn7k" Nov 01 20:38:03.015658 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.015632 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8jbj\" (UniqueName: \"kubernetes.io/projected/b790dce9-8a9b-4012-91c1-da328a2199be-kube-api-access-p8jbj\") pod \"authorino-7f66647665-4cn7k\" (UID: \"b790dce9-8a9b-4012-91c1-da328a2199be\") " pod="kuadrant-system/authorino-7f66647665-4cn7k" Nov 01 20:38:03.059280 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.059208 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f66647665-4cn7k"] Nov 01 20:38:03.116961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.116931 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8jbj\" (UniqueName: \"kubernetes.io/projected/b790dce9-8a9b-4012-91c1-da328a2199be-kube-api-access-p8jbj\") pod \"authorino-7f66647665-4cn7k\" (UID: \"b790dce9-8a9b-4012-91c1-da328a2199be\") " pod="kuadrant-system/authorino-7f66647665-4cn7k" Nov 01 20:38:03.177012 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.176974 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8jbj\" (UniqueName: \"kubernetes.io/projected/b790dce9-8a9b-4012-91c1-da328a2199be-kube-api-access-p8jbj\") pod \"authorino-7f66647665-4cn7k\" (UID: \"b790dce9-8a9b-4012-91c1-da328a2199be\") " pod="kuadrant-system/authorino-7f66647665-4cn7k" Nov 01 20:38:03.316004 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.315968 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f66647665-4cn7k" Nov 01 20:38:03.347425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.347241 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qr74p"] Nov 01 20:38:03.351115 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:38:03.351051 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6a1e955_1da5_420c_86ac_99dc980bda42.slice/crio-469bffdf8d8d861dd28e4659d3e2b7ea22ffe348167219a765a377c9d89be21c WatchSource:0}: Error finding container 469bffdf8d8d861dd28e4659d3e2b7ea22ffe348167219a765a377c9d89be21c: Status 404 returned error can't find the container with id 469bffdf8d8d861dd28e4659d3e2b7ea22ffe348167219a765a377c9d89be21c Nov 01 20:38:03.496430 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.496394 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f66647665-4cn7k"] Nov 01 20:38:03.527808 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.527776 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b8bd54887-hxpbx"] Nov 01 20:38:03.530602 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.530581 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:38:03.584148 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.583948 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 20:38:03.587600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.587572 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b8bd54887-hxpbx"] Nov 01 20:38:03.620168 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.620137 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74sv\" (UniqueName: \"kubernetes.io/projected/1a19ee76-5192-4239-a391-5c7b27b21c4a-kube-api-access-g74sv\") pod \"authorino-7b8bd54887-hxpbx\" (UID: \"1a19ee76-5192-4239-a391-5c7b27b21c4a\") " pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:38:03.620306 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.620205 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a19ee76-5192-4239-a391-5c7b27b21c4a-tls-cert\") pod \"authorino-7b8bd54887-hxpbx\" (UID: \"1a19ee76-5192-4239-a391-5c7b27b21c4a\") " pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:38:03.688599 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.688577 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f66647665-4cn7k"] Nov 01 20:38:03.690488 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:38:03.690458 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb790dce9_8a9b_4012_91c1_da328a2199be.slice/crio-8a90092fd8fc16741a87bba4d8faa35e9a9722bbded2cf120969bf4733d0e54c WatchSource:0}: Error finding container 8a90092fd8fc16741a87bba4d8faa35e9a9722bbded2cf120969bf4733d0e54c: Status 404 returned error can't find the container with id 8a90092fd8fc16741a87bba4d8faa35e9a9722bbded2cf120969bf4733d0e54c Nov 01 20:38:03.721577 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.721538 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a19ee76-5192-4239-a391-5c7b27b21c4a-tls-cert\") pod \"authorino-7b8bd54887-hxpbx\" (UID: \"1a19ee76-5192-4239-a391-5c7b27b21c4a\") " pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:38:03.721721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.721596 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74sv\" (UniqueName: \"kubernetes.io/projected/1a19ee76-5192-4239-a391-5c7b27b21c4a-kube-api-access-g74sv\") pod \"authorino-7b8bd54887-hxpbx\" (UID: \"1a19ee76-5192-4239-a391-5c7b27b21c4a\") " pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:38:03.724100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.724061 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a19ee76-5192-4239-a391-5c7b27b21c4a-tls-cert\") pod \"authorino-7b8bd54887-hxpbx\" (UID: \"1a19ee76-5192-4239-a391-5c7b27b21c4a\") " pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:38:03.777642 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.777612 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74sv\" (UniqueName: \"kubernetes.io/projected/1a19ee76-5192-4239-a391-5c7b27b21c4a-kube-api-access-g74sv\") pod \"authorino-7b8bd54887-hxpbx\" (UID: \"1a19ee76-5192-4239-a391-5c7b27b21c4a\") " pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:38:03.841394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.841331 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:38:03.990057 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.990021 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f66647665-4cn7k" event={"ID":"b790dce9-8a9b-4012-91c1-da328a2199be","Type":"ContainerStarted","Data":"8a90092fd8fc16741a87bba4d8faa35e9a9722bbded2cf120969bf4733d0e54c"} Nov 01 20:38:03.991354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:03.991326 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" event={"ID":"a6a1e955-1da5-420c-86ac-99dc980bda42","Type":"ContainerStarted","Data":"469bffdf8d8d861dd28e4659d3e2b7ea22ffe348167219a765a377c9d89be21c"} Nov 01 20:38:04.216824 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:04.216792 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b8bd54887-hxpbx"] Nov 01 20:38:04.218988 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:38:04.218961 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a19ee76_5192_4239_a391_5c7b27b21c4a.slice/crio-6fa922c3c1ee6d7799404494b85bd0f0d749ef234e47489d4234ad6919c44e54 WatchSource:0}: Error finding container 6fa922c3c1ee6d7799404494b85bd0f0d749ef234e47489d4234ad6919c44e54: Status 404 returned error can't find the container with id 6fa922c3c1ee6d7799404494b85bd0f0d749ef234e47489d4234ad6919c44e54 Nov 01 20:38:04.997774 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:04.997739 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" event={"ID":"1a19ee76-5192-4239-a391-5c7b27b21c4a","Type":"ContainerStarted","Data":"6fa922c3c1ee6d7799404494b85bd0f0d749ef234e47489d4234ad6919c44e54"} Nov 01 20:38:04.999434 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:04.999403 4477 generic.go:356] "Generic (PLEG): container finished" podID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerID="1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e" exitCode=0 Nov 01 20:38:04.999570 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:04.999462 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" event={"ID":"80d7cf7c-22e5-4201-852e-98b1b0944656","Type":"ContainerDied","Data":"1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e"} Nov 01 20:38:04.999881 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:04.999849 4477 scope.go:122] "RemoveContainer" containerID="1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e" Nov 01 20:38:05.000986 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:05.000956 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:38:05.001056 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:05.001028 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:38:05.001119 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:05.001061 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:38:06.010761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.008640 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f66647665-4cn7k" event={"ID":"b790dce9-8a9b-4012-91c1-da328a2199be","Type":"ContainerStarted","Data":"c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56"} Nov 01 20:38:06.010761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.008844 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7f66647665-4cn7k" podUID="b790dce9-8a9b-4012-91c1-da328a2199be" containerName="authorino" containerID="cri-o://c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56" gracePeriod=30 Nov 01 20:38:06.040101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.030419 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" event={"ID":"a6a1e955-1da5-420c-86ac-99dc980bda42","Type":"ContainerStarted","Data":"b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a"} Nov 01 20:38:06.040101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.031718 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" podUID="a6a1e955-1da5-420c-86ac-99dc980bda42" containerName="authorino" containerID="cri-o://b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a" gracePeriod=30 Nov 01 20:38:06.043027 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.042538 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" event={"ID":"80d7cf7c-22e5-4201-852e-98b1b0944656","Type":"ContainerStarted","Data":"9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51"} Nov 01 20:38:06.046447 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.046420 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" event={"ID":"1a19ee76-5192-4239-a391-5c7b27b21c4a","Type":"ContainerStarted","Data":"d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0"} Nov 01 20:38:06.107823 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.107797 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:38:06.107993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.107972 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z2jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Nov 01 20:38:06.108040 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.108018 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Nov 01 20:38:06.141132 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.141048 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7f66647665-4cn7k" podStartSLOduration=2.7423004090000003 podStartE2EDuration="4.141028432s" totalImagesPullingTime="1.398728023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:06.138344868 +0000 UTC m=+439.788396441" watchObservedRunningTime="2026-11-01 20:38:06.141028432 +0000 UTC m=+439.791079978" Nov 01 20:38:06.253764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.253712 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" podStartSLOduration=1.820388243 podStartE2EDuration="3.25369697s" totalImagesPullingTime="1.433308727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:06.251815264 +0000 UTC m=+439.901866845" watchObservedRunningTime="2026-11-01 20:38:06.25369697 +0000 UTC m=+439.903748518" Nov 01 20:38:06.372717 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.372695 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" Nov 01 20:38:06.376464 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.376445 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f66647665-4cn7k" Nov 01 20:38:06.450607 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.450569 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a1e955-1da5-420c-86ac-99dc980bda42-kube-api-access-vw727\" (UniqueName: \"kubernetes.io/projected/a6a1e955-1da5-420c-86ac-99dc980bda42-kube-api-access-vw727\") pod \"a6a1e955-1da5-420c-86ac-99dc980bda42\" (UID: \"a6a1e955-1da5-420c-86ac-99dc980bda42\") " Nov 01 20:38:06.450793 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.450679 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b790dce9-8a9b-4012-91c1-da328a2199be-kube-api-access-p8jbj\" (UniqueName: \"kubernetes.io/projected/b790dce9-8a9b-4012-91c1-da328a2199be-kube-api-access-p8jbj\") pod \"b790dce9-8a9b-4012-91c1-da328a2199be\" (UID: \"b790dce9-8a9b-4012-91c1-da328a2199be\") " Nov 01 20:38:06.453020 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.452988 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a1e955-1da5-420c-86ac-99dc980bda42-kube-api-access-vw727" pod "a6a1e955-1da5-420c-86ac-99dc980bda42" (UID: "a6a1e955-1da5-420c-86ac-99dc980bda42"). InnerVolumeSpecName "kube-api-access-vw727". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:06.453249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.453194 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b790dce9-8a9b-4012-91c1-da328a2199be-kube-api-access-p8jbj" pod "b790dce9-8a9b-4012-91c1-da328a2199be" (UID: "b790dce9-8a9b-4012-91c1-da328a2199be"). InnerVolumeSpecName "kube-api-access-p8jbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:06.496357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.496300 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" podStartSLOduration=2.832250802 podStartE2EDuration="4.496285137s" totalImagesPullingTime="1.664034335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:06.494984517 +0000 UTC m=+440.145036061" watchObservedRunningTime="2026-11-01 20:38:06.496285137 +0000 UTC m=+440.146336682" Nov 01 20:38:06.551324 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.551263 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8jbj\" (UniqueName: \"kubernetes.io/projected/b790dce9-8a9b-4012-91c1-da328a2199be-kube-api-access-p8jbj\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:06.551324 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:06.551289 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw727\" (UniqueName: \"kubernetes.io/projected/a6a1e955-1da5-420c-86ac-99dc980bda42-kube-api-access-vw727\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:06.571144 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:38:06.571103 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:38:07.053332 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.053297 4477 generic.go:356] "Generic (PLEG): container finished" podID="b790dce9-8a9b-4012-91c1-da328a2199be" containerID="c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56" exitCode=0 Nov 01 20:38:07.053759 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.053343 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f66647665-4cn7k" Nov 01 20:38:07.053759 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.053398 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f66647665-4cn7k" event={"ID":"b790dce9-8a9b-4012-91c1-da328a2199be","Type":"ContainerDied","Data":"c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56"} Nov 01 20:38:07.053759 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.053446 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f66647665-4cn7k" event={"ID":"b790dce9-8a9b-4012-91c1-da328a2199be","Type":"ContainerDied","Data":"8a90092fd8fc16741a87bba4d8faa35e9a9722bbded2cf120969bf4733d0e54c"} Nov 01 20:38:07.053759 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.053466 4477 scope.go:122] "RemoveContainer" containerID="c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56" Nov 01 20:38:07.055171 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.055109 4477 generic.go:356] "Generic (PLEG): container finished" podID="a6a1e955-1da5-420c-86ac-99dc980bda42" containerID="b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a" exitCode=0 Nov 01 20:38:07.055171 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.055156 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" Nov 01 20:38:07.055354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.055212 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" event={"ID":"a6a1e955-1da5-420c-86ac-99dc980bda42","Type":"ContainerDied","Data":"b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a"} Nov 01 20:38:07.055354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.055240 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qr74p" event={"ID":"a6a1e955-1da5-420c-86ac-99dc980bda42","Type":"ContainerDied","Data":"469bffdf8d8d861dd28e4659d3e2b7ea22ffe348167219a765a377c9d89be21c"} Nov 01 20:38:07.062898 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.062880 4477 scope.go:122] "RemoveContainer" containerID="c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56" Nov 01 20:38:07.063200 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:38:07.063177 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56\": container with ID starting with c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56 not found: ID does not exist" containerID="c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56" Nov 01 20:38:07.063296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.063212 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56"} err="failed to get container status \"c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56\": rpc error: code = NotFound desc = could not find container \"c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56\": container with ID starting with c23501f9f1ab11d0c901d9107773ca354da792c0dbf38445e49791f38c533d56 not found: ID does not exist" Nov 01 20:38:07.063296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.063230 4477 scope.go:122] "RemoveContainer" containerID="b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a" Nov 01 20:38:07.071135 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.071117 4477 scope.go:122] "RemoveContainer" containerID="b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a" Nov 01 20:38:07.071424 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:38:07.071395 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a\": container with ID starting with b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a not found: ID does not exist" containerID="b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a" Nov 01 20:38:07.071522 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.071429 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a"} err="failed to get container status \"b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a\": rpc error: code = NotFound desc = could not find container \"b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a\": container with ID starting with b6e33d6b6d6f64133e4ea3dccfd9a5126e4f770ffacca4285c950c873b383d4a not found: ID does not exist" Nov 01 20:38:07.107397 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.107375 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z2jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Nov 01 20:38:07.107514 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.107429 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Nov 01 20:38:07.112613 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.112590 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m74c5"] Nov 01 20:38:07.113010 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.112997 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b790dce9-8a9b-4012-91c1-da328a2199be" containerName="authorino" Nov 01 20:38:07.113056 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.113011 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b790dce9-8a9b-4012-91c1-da328a2199be" containerName="authorino" Nov 01 20:38:07.113056 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.113020 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a6a1e955-1da5-420c-86ac-99dc980bda42" containerName="authorino" Nov 01 20:38:07.113056 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.113024 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a1e955-1da5-420c-86ac-99dc980bda42" containerName="authorino" Nov 01 20:38:07.113221 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.113095 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b790dce9-8a9b-4012-91c1-da328a2199be" containerName="authorino" Nov 01 20:38:07.113221 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.113114 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a1e955-1da5-420c-86ac-99dc980bda42" containerName="authorino" Nov 01 20:38:07.139491 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.139462 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:07.155646 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.155617 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8ql\" (UniqueName: \"kubernetes.io/projected/b3017bf6-3e29-4780-9053-6853d3329613-kube-api-access-bj8ql\") pod \"llmisvc-controller-manager-6466c554d8-m74c5\" (UID: \"b3017bf6-3e29-4780-9053-6853d3329613\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:07.155866 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.155843 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3017bf6-3e29-4780-9053-6853d3329613-cert\") pod \"llmisvc-controller-manager-6466c554d8-m74c5\" (UID: \"b3017bf6-3e29-4780-9053-6853d3329613\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:07.171569 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.171536 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m74c5"] Nov 01 20:38:07.240213 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.240182 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f66647665-4cn7k"] Nov 01 20:38:07.241986 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.241964 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f66647665-4cn7k"] Nov 01 20:38:07.256456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.256431 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8ql\" (UniqueName: \"kubernetes.io/projected/b3017bf6-3e29-4780-9053-6853d3329613-kube-api-access-bj8ql\") pod \"llmisvc-controller-manager-6466c554d8-m74c5\" (UID: \"b3017bf6-3e29-4780-9053-6853d3329613\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:07.256530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.256492 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3017bf6-3e29-4780-9053-6853d3329613-cert\") pod \"llmisvc-controller-manager-6466c554d8-m74c5\" (UID: \"b3017bf6-3e29-4780-9053-6853d3329613\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:07.259176 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.259157 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3017bf6-3e29-4780-9053-6853d3329613-cert\") pod \"llmisvc-controller-manager-6466c554d8-m74c5\" (UID: \"b3017bf6-3e29-4780-9053-6853d3329613\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:07.312805 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.312776 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8ql\" (UniqueName: \"kubernetes.io/projected/b3017bf6-3e29-4780-9053-6853d3329613-kube-api-access-bj8ql\") pod \"llmisvc-controller-manager-6466c554d8-m74c5\" (UID: \"b3017bf6-3e29-4780-9053-6853d3329613\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:07.343803 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.343771 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qr74p"] Nov 01 20:38:07.346753 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.346728 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qr74p"] Nov 01 20:38:07.451465 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.451432 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:07.836115 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:07.833990 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m74c5"] Nov 01 20:38:07.841971 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:38:07.841940 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3017bf6_3e29_4780_9053_6853d3329613.slice/crio-9e08090f914caa5156a693cf5a5723a65d79b02bc8008804553a32e9c4485682 WatchSource:0}: Error finding container 9e08090f914caa5156a693cf5a5723a65d79b02bc8008804553a32e9c4485682: Status 404 returned error can't find the container with id 9e08090f914caa5156a693cf5a5723a65d79b02bc8008804553a32e9c4485682 Nov 01 20:38:08.061981 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:08.061952 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" event={"ID":"b3017bf6-3e29-4780-9053-6853d3329613","Type":"ContainerStarted","Data":"9e08090f914caa5156a693cf5a5723a65d79b02bc8008804553a32e9c4485682"} Nov 01 20:38:08.108358 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:08.108284 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z2jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Nov 01 20:38:08.108358 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:08.108331 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Nov 01 20:38:08.502452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:08.502368 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a1e955-1da5-420c-86ac-99dc980bda42" path="/var/lib/kubelet/pods/a6a1e955-1da5-420c-86ac-99dc980bda42/volumes" Nov 01 20:38:08.502744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:08.502729 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b790dce9-8a9b-4012-91c1-da328a2199be" path="/var/lib/kubelet/pods/b790dce9-8a9b-4012-91c1-da328a2199be/volumes" Nov 01 20:38:09.071155 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:09.071125 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" event={"ID":"b3017bf6-3e29-4780-9053-6853d3329613","Type":"ContainerStarted","Data":"45c2826687be9b839285bfad6aaf70938ecf3a46ec1f364e4bc361297118115b"} Nov 01 20:38:09.071532 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:09.071197 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:09.107780 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:09.107749 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z2jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Nov 01 20:38:09.107923 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:09.107795 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Nov 01 20:38:09.188412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:09.188361 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" podStartSLOduration=1.434126838 podStartE2EDuration="2.188342477s" totalImagesPullingTime="754.215639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:09.187043498 +0000 UTC m=+442.837095062" watchObservedRunningTime="2026-11-01 20:38:09.188342477 +0000 UTC m=+442.838394025" Nov 01 20:38:10.107411 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:10.107382 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z2jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Nov 01 20:38:10.107791 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:10.107428 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Nov 01 20:38:11.137421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:11.137385 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:38:11.138632 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:11.138611 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:38:11.139269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:11.139253 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:38:36.552888 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:38:36.552846 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:38:40.078785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.078756 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m74c5" Nov 01 20:38:40.243837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.243808 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4"] Nov 01 20:38:40.244018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.243995 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" podUID="32462b71-2673-4121-902f-a1167c23eaa9" containerName="manager" containerID="cri-o://db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5" gracePeriod=30 Nov 01 20:38:40.492343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.492318 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:38:40.626687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.626602 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32462b71-2673-4121-902f-a1167c23eaa9-cert\" (UniqueName: \"kubernetes.io/secret/32462b71-2673-4121-902f-a1167c23eaa9-cert\") pod \"32462b71-2673-4121-902f-a1167c23eaa9\" (UID: \"32462b71-2673-4121-902f-a1167c23eaa9\") " Nov 01 20:38:40.626687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.626662 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32462b71-2673-4121-902f-a1167c23eaa9-kube-api-access-ldvwq\" (UniqueName: \"kubernetes.io/projected/32462b71-2673-4121-902f-a1167c23eaa9-kube-api-access-ldvwq\") pod \"32462b71-2673-4121-902f-a1167c23eaa9\" (UID: \"32462b71-2673-4121-902f-a1167c23eaa9\") " Nov 01 20:38:40.629693 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.629658 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32462b71-2673-4121-902f-a1167c23eaa9-cert" pod "32462b71-2673-4121-902f-a1167c23eaa9" (UID: "32462b71-2673-4121-902f-a1167c23eaa9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:38:40.629835 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.629785 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32462b71-2673-4121-902f-a1167c23eaa9-kube-api-access-ldvwq" pod "32462b71-2673-4121-902f-a1167c23eaa9" (UID: "32462b71-2673-4121-902f-a1167c23eaa9"). InnerVolumeSpecName "kube-api-access-ldvwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:40.727831 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.727796 4477 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32462b71-2673-4121-902f-a1167c23eaa9-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:40.727831 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:40.727824 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldvwq\" (UniqueName: \"kubernetes.io/projected/32462b71-2673-4121-902f-a1167c23eaa9-kube-api-access-ldvwq\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:41.252284 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.252249 4477 generic.go:356] "Generic (PLEG): container finished" podID="32462b71-2673-4121-902f-a1167c23eaa9" containerID="db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5" exitCode=0 Nov 01 20:38:41.252738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.252299 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" Nov 01 20:38:41.252738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.252332 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" event={"ID":"32462b71-2673-4121-902f-a1167c23eaa9","Type":"ContainerDied","Data":"db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5"} Nov 01 20:38:41.252738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.252402 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4" event={"ID":"32462b71-2673-4121-902f-a1167c23eaa9","Type":"ContainerDied","Data":"658f9e5a5daaad2301f9f061860f4c5ceca5e4f5c70a68291d0214c6f5d898bd"} Nov 01 20:38:41.252738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.252422 4477 scope.go:122] "RemoveContainer" containerID="db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5" Nov 01 20:38:41.262149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.262135 4477 scope.go:122] "RemoveContainer" containerID="db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5" Nov 01 20:38:41.262397 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:38:41.262382 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5\": container with ID starting with db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5 not found: ID does not exist" containerID="db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5" Nov 01 20:38:41.262452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.262404 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5"} err="failed to get container status \"db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5\": rpc error: code = NotFound desc = could not find container \"db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5\": container with ID starting with db121d5af68f55298af6a4e5ed0accd6438cbf151964aac91ad3087ca999fdd5 not found: ID does not exist" Nov 01 20:38:41.415151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.415116 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4"] Nov 01 20:38:41.464656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:41.464614 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-cnnc4"] Nov 01 20:38:42.502826 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:38:42.502792 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32462b71-2673-4121-902f-a1167c23eaa9" path="/var/lib/kubelet/pods/32462b71-2673-4121-902f-a1167c23eaa9/volumes" Nov 01 20:39:06.551820 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:39:06.551739 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:39:07.802516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:39:07.802468 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:39:20.575182 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:39:20.575144 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:39:32.632262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:39:32.632227 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:39:36.553406 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:39:36.553360 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:39:45.458821 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:39:45.458779 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:40:01.925102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:01.925051 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b8bd54887-hxpbx"] Nov 01 20:40:01.925571 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:01.925284 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" podUID="1a19ee76-5192-4239-a391-5c7b27b21c4a" containerName="authorino" containerID="cri-o://d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0" gracePeriod=30 Nov 01 20:40:02.181500 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.181428 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:40:02.359423 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.359389 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a19ee76-5192-4239-a391-5c7b27b21c4a-tls-cert\" (UniqueName: \"kubernetes.io/secret/1a19ee76-5192-4239-a391-5c7b27b21c4a-tls-cert\") pod \"1a19ee76-5192-4239-a391-5c7b27b21c4a\" (UID: \"1a19ee76-5192-4239-a391-5c7b27b21c4a\") " Nov 01 20:40:02.359620 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.359434 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a19ee76-5192-4239-a391-5c7b27b21c4a-kube-api-access-g74sv\" (UniqueName: \"kubernetes.io/projected/1a19ee76-5192-4239-a391-5c7b27b21c4a-kube-api-access-g74sv\") pod \"1a19ee76-5192-4239-a391-5c7b27b21c4a\" (UID: \"1a19ee76-5192-4239-a391-5c7b27b21c4a\") " Nov 01 20:40:02.361702 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.361673 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a19ee76-5192-4239-a391-5c7b27b21c4a-kube-api-access-g74sv" pod "1a19ee76-5192-4239-a391-5c7b27b21c4a" (UID: "1a19ee76-5192-4239-a391-5c7b27b21c4a"). InnerVolumeSpecName "kube-api-access-g74sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:02.371725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.371694 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a19ee76-5192-4239-a391-5c7b27b21c4a-tls-cert" pod "1a19ee76-5192-4239-a391-5c7b27b21c4a" (UID: "1a19ee76-5192-4239-a391-5c7b27b21c4a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:40:02.460760 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.460678 4477 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a19ee76-5192-4239-a391-5c7b27b21c4a-tls-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:02.460760 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.460707 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g74sv\" (UniqueName: \"kubernetes.io/projected/1a19ee76-5192-4239-a391-5c7b27b21c4a-kube-api-access-g74sv\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:02.698849 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.698809 4477 generic.go:356] "Generic (PLEG): container finished" podID="1a19ee76-5192-4239-a391-5c7b27b21c4a" containerID="d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0" exitCode=0 Nov 01 20:40:02.699059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.698861 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" Nov 01 20:40:02.699059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.698903 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" event={"ID":"1a19ee76-5192-4239-a391-5c7b27b21c4a","Type":"ContainerDied","Data":"d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0"} Nov 01 20:40:02.699059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.698968 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8bd54887-hxpbx" event={"ID":"1a19ee76-5192-4239-a391-5c7b27b21c4a","Type":"ContainerDied","Data":"6fa922c3c1ee6d7799404494b85bd0f0d749ef234e47489d4234ad6919c44e54"} Nov 01 20:40:02.699059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.698993 4477 scope.go:122] "RemoveContainer" containerID="d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0" Nov 01 20:40:02.709012 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.708990 4477 scope.go:122] "RemoveContainer" containerID="d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0" Nov 01 20:40:02.709358 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:40:02.709334 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0\": container with ID starting with d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0 not found: ID does not exist" containerID="d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0" Nov 01 20:40:02.709436 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.709371 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0"} err="failed to get container status \"d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0\": rpc error: code = NotFound desc = could not find container \"d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0\": container with ID starting with d47a8ae8bb0d49fe5b79c1070dba9e98778fd24457ee88a536c8ebff4cc890a0 not found: ID does not exist" Nov 01 20:40:02.861140 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.861107 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b8bd54887-hxpbx"] Nov 01 20:40:02.864806 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:02.864782 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b8bd54887-hxpbx"] Nov 01 20:40:04.503313 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.503283 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a19ee76-5192-4239-a391-5c7b27b21c4a" path="/var/lib/kubelet/pods/1a19ee76-5192-4239-a391-5c7b27b21c4a/volumes" Nov 01 20:40:04.516984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.516947 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk"] Nov 01 20:40:04.517361 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.517348 4477 cpu_manager.go:414] "containerMap: removing container" podUID="1a19ee76-5192-4239-a391-5c7b27b21c4a" containerName="authorino" Nov 01 20:40:04.517413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.517362 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a19ee76-5192-4239-a391-5c7b27b21c4a" containerName="authorino" Nov 01 20:40:04.517413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.517382 4477 cpu_manager.go:414] "containerMap: removing container" podUID="32462b71-2673-4121-902f-a1167c23eaa9" containerName="manager" Nov 01 20:40:04.517413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.517387 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32462b71-2673-4121-902f-a1167c23eaa9" containerName="manager" Nov 01 20:40:04.517519 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.517460 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a19ee76-5192-4239-a391-5c7b27b21c4a" containerName="authorino" Nov 01 20:40:04.517519 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.517470 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="32462b71-2673-4121-902f-a1167c23eaa9" containerName="manager" Nov 01 20:40:04.522199 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.522173 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.619554 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.619522 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk"] Nov 01 20:40:04.680299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680268 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.680485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680319 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.680485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680404 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.680559 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680495 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-kube-api-access-59b2g\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.680559 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680546 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.680627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680571 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.680627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680597 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57227b86-8401-4e61-8f01-daf8cdfbd753-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.680721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680706 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.680758 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.680734 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781268 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781305 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781338 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781621 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781376 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781621 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781397 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781621 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781428 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-kube-api-access-59b2g\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781621 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781457 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781621 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781483 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.781621 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781520 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57227b86-8401-4e61-8f01-daf8cdfbd753-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.782033 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781748 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.782033 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.781821 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.782161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.782136 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.782288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.782269 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57227b86-8401-4e61-8f01-daf8cdfbd753-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.782512 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.782445 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.784039 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.784017 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.784552 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.784529 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.837650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.837621 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-kube-api-access-59b2g\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:04.837740 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:04.837686 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-22bjk\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:05.136455 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:05.136420 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:05.565113 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:05.565043 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk"] Nov 01 20:40:05.568019 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:40:05.567980 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57227b86_8401_4e61_8f01_daf8cdfbd753.slice/crio-bd6d38acc3ba54223465de316700328f4669cd9abbc6ffb618edc5d73211d681 WatchSource:0}: Error finding container bd6d38acc3ba54223465de316700328f4669cd9abbc6ffb618edc5d73211d681: Status 404 returned error can't find the container with id bd6d38acc3ba54223465de316700328f4669cd9abbc6ffb618edc5d73211d681 Nov 01 20:40:05.570232 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:05.570196 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:40:05.570303 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:05.570273 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:40:05.570343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:05.570303 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:40:05.716953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:05.716917 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" event={"ID":"57227b86-8401-4e61-8f01-daf8cdfbd753","Type":"ContainerStarted","Data":"e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95"} Nov 01 20:40:05.717135 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:05.716969 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" event={"ID":"57227b86-8401-4e61-8f01-daf8cdfbd753","Type":"ContainerStarted","Data":"bd6d38acc3ba54223465de316700328f4669cd9abbc6ffb618edc5d73211d681"} Nov 01 20:40:05.833285 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:05.833193 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podStartSLOduration=1.83317942 podStartE2EDuration="1.83317942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:05.831612238 +0000 UTC m=+559.481663785" watchObservedRunningTime="2026-11-01 20:40:05.83317942 +0000 UTC m=+559.483230964" Nov 01 20:40:06.138373 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:06.138270 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:06.139971 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:06.139943 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-22bjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:40:06.140157 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:06.139991 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 01 20:40:06.570727 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:40:06.570686 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:40:07.138101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:07.137513 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-22bjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:40:07.138101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:07.137580 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 01 20:40:08.138589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:08.138542 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-22bjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:40:08.139161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:08.138610 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 01 20:40:09.137178 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:09.137125 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-22bjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:40:09.137502 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:09.137454 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 01 20:40:10.138525 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:10.138480 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-22bjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:40:10.139125 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:10.138574 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 01 20:40:11.138102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:11.136870 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-22bjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:40:11.138102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:11.136940 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 01 20:40:12.136972 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:12.136933 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-22bjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:40:12.137472 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:12.136990 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 01 20:40:13.817423 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:13.817390 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:13.817932 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:13.817753 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:13.819099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:13.819057 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:40:34.759388 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.759306 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt"] Nov 01 20:40:34.762803 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.762784 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865489 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865535 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865620 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865655 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865687 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865824 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqg26\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-kube-api-access-kqg26\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865905 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865873 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865940 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865930 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.865983 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.865957 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.866929 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.866911 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt"] Nov 01 20:40:34.967247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967211 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967251 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967267 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967305 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967350 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqg26\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-kube-api-access-kqg26\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967382 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967410 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967434 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967503 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967826 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967950 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967907 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.967950 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.967914 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.968132 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.968110 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.968206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.968169 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.969874 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.969851 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:34.970015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:34.969995 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:35.023688 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.023609 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:35.023688 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.023615 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqg26\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-kube-api-access-kqg26\") pod \"maas-default-gateway-openshift-default-58c6585447-pcwvt\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:35.076549 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.076525 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:35.499373 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.499344 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt"] Nov 01 20:40:35.501898 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:40:35.501869 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67e64c1_5c69_473e_8e56_d9c6be9c6d2e.slice/crio-df8aa6e8754fc107f322c4fb38a625a0d8d5f6848fecf117e7c712ff1f0fcb25 WatchSource:0}: Error finding container df8aa6e8754fc107f322c4fb38a625a0d8d5f6848fecf117e7c712ff1f0fcb25: Status 404 returned error can't find the container with id df8aa6e8754fc107f322c4fb38a625a0d8d5f6848fecf117e7c712ff1f0fcb25 Nov 01 20:40:35.504150 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.504112 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:40:35.504252 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.504205 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:40:35.504311 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.504252 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:40:35.911650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.911612 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" event={"ID":"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e","Type":"ContainerStarted","Data":"2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24"} Nov 01 20:40:35.912056 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:35.911690 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" event={"ID":"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e","Type":"ContainerStarted","Data":"df8aa6e8754fc107f322c4fb38a625a0d8d5f6848fecf117e7c712ff1f0fcb25"} Nov 01 20:40:36.076967 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:36.076931 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:36.078429 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:36.078408 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pcwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:40:36.078491 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:36.078460 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:40:36.564248 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:40:36.564211 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:40:37.077619 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:37.077593 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pcwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:40:37.077992 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:37.077653 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:40:38.077849 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:38.077813 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pcwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:40:38.078295 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:38.077879 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:40:39.077454 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:39.077408 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pcwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:40:39.077619 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:39.077486 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:40:40.077020 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:40.076974 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pcwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:40:40.077464 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:40.077040 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:40:41.077187 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:41.077155 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pcwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:40:41.077593 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:41.077208 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:40:42.077531 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:42.077488 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pcwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:40:42.077958 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:42.077550 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:40:43.741317 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:43.741285 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:43.741874 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:43.741596 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:43.742413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:43.742393 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:40:43.913621 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:40:43.913549 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podStartSLOduration=9.913528482 podStartE2EDuration="9.913528482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:36.038460095 +0000 UTC m=+589.688511641" watchObservedRunningTime="2026-11-01 20:40:43.913528482 +0000 UTC m=+597.563580033" Nov 01 20:41:06.554230 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:41:06.554188 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:41:24.155798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:24.155751 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:41:33.032768 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:33.032720 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:41:36.552233 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:41:36.552182 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:41:43.369711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:43.369674 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:41:51.274728 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:51.274691 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:41:55.153850 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.153820 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw"] Nov 01 20:41:55.157625 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.157601 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.226771 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.226735 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-metrics-certs\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.226941 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.226779 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgzzc\" (UniqueName: \"kubernetes.io/projected/68baa7a8-a64f-4f0b-84cc-47bb018042c7-kube-api-access-qgzzc\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.226941 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.226903 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68baa7a8-a64f-4f0b-84cc-47bb018042c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.227058 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.227038 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.266135 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.266069 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-1a1e9e-serving-cert\"" Nov 01 20:41:55.266323 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.266173 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 20:41:55.266323 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.266181 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cms9v\"" Nov 01 20:41:55.268540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.268517 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw"] Nov 01 20:41:55.327613 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.327569 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.327802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.327662 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-metrics-certs\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.327802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.327685 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgzzc\" (UniqueName: \"kubernetes.io/projected/68baa7a8-a64f-4f0b-84cc-47bb018042c7-kube-api-access-qgzzc\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.327802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.327709 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68baa7a8-a64f-4f0b-84cc-47bb018042c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.327802 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:41:55.327735 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-serving-cert: secret "maas-api-e2e-worker-w4-1a1e9e-serving-cert" not found Nov 01 20:41:55.328010 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:41:55.327812 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls podName:68baa7a8-a64f-4f0b-84cc-47bb018042c7 nodeName:}" failed. No retries permitted until 2026-11-01 20:41:55.827791339 +0000 UTC m=+669.477842863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls") pod "maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" (UID: "68baa7a8-a64f-4f0b-84cc-47bb018042c7") : secret "maas-api-e2e-worker-w4-1a1e9e-serving-cert" not found Nov 01 20:41:55.328471 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.328417 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68baa7a8-a64f-4f0b-84cc-47bb018042c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.330355 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.330331 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-metrics-certs\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.384769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.384733 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgzzc\" (UniqueName: \"kubernetes.io/projected/68baa7a8-a64f-4f0b-84cc-47bb018042c7-kube-api-access-qgzzc\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.833616 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.833584 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:55.836269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:55.836244 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:56.070323 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:56.070290 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:41:56.450068 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:56.449991 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw"] Nov 01 20:41:56.453001 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:41:56.452960 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68baa7a8_a64f_4f0b_84cc_47bb018042c7.slice/crio-8a79f4497b9b4978120fd6e265f5488869144fdbaa8bddc12b6c86cf358ab9df WatchSource:0}: Error finding container 8a79f4497b9b4978120fd6e265f5488869144fdbaa8bddc12b6c86cf358ab9df: Status 404 returned error can't find the container with id 8a79f4497b9b4978120fd6e265f5488869144fdbaa8bddc12b6c86cf358ab9df Nov 01 20:41:56.455046 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:56.455025 4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:41:57.378452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:41:57.378418 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" event={"ID":"68baa7a8-a64f-4f0b-84cc-47bb018042c7","Type":"ContainerStarted","Data":"8a79f4497b9b4978120fd6e265f5488869144fdbaa8bddc12b6c86cf358ab9df"} Nov 01 20:42:00.410684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.410651 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" event={"ID":"68baa7a8-a64f-4f0b-84cc-47bb018042c7","Type":"ContainerStarted","Data":"abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162"} Nov 01 20:42:00.411136 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.410792 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:42:00.514014 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.513976 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4"] Nov 01 20:42:00.517743 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.517718 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.563209 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.563151 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" podStartSLOduration=2.034834924 podStartE2EDuration="5.563135786s" totalImagesPullingTime="3.528300862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:42:00.529039975 +0000 UTC m=+674.179091538" watchObservedRunningTime="2026-11-01 20:42:00.563135786 +0000 UTC m=+674.213187331" Nov 01 20:42:00.565990 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.565970 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4"] Nov 01 20:42:00.569911 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.569880 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-c9ae21-serving-cert\"" Nov 01 20:42:00.580471 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.580444 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.580645 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.580617 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347f23c-9e24-472c-b4e0-0da2661b8f6c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.580846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.580824 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.580928 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.580864 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8gq\" (UniqueName: \"kubernetes.io/projected/0347f23c-9e24-472c-b4e0-0da2661b8f6c-kube-api-access-hk8gq\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.682098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.681992 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.682098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.682034 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8gq\" (UniqueName: \"kubernetes.io/projected/0347f23c-9e24-472c-b4e0-0da2661b8f6c-kube-api-access-hk8gq\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.682098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.682067 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.682416 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:42:00.682185 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-serving-cert: secret "maas-api-e2e-worker-w5-c9ae21-serving-cert" not found Nov 01 20:42:00.682416 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:42:00.682243 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls podName:0347f23c-9e24-472c-b4e0-0da2661b8f6c nodeName:}" failed. No retries permitted until 2026-11-01 20:42:01.182224147 +0000 UTC m=+674.832275670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls") pod "maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" (UID: "0347f23c-9e24-472c-b4e0-0da2661b8f6c") : secret "maas-api-e2e-worker-w5-c9ae21-serving-cert" not found Nov 01 20:42:00.682416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.682331 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347f23c-9e24-472c-b4e0-0da2661b8f6c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.683398 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.683360 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347f23c-9e24-472c-b4e0-0da2661b8f6c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.684986 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.684959 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:00.739537 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:00.739498 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8gq\" (UniqueName: \"kubernetes.io/projected/0347f23c-9e24-472c-b4e0-0da2661b8f6c-kube-api-access-hk8gq\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:01.186443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:01.186404 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:01.189050 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:01.189025 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:01.432695 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:01.432662 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:01.820429 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:01.820338 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4"] Nov 01 20:42:01.822739 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:42:01.822708 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0347f23c_9e24_472c_b4e0_0da2661b8f6c.slice/crio-0b15d8e8b7f2a68132d8b2693ca0cac1850d12c0f9e84129afa363a163a2a0a0 WatchSource:0}: Error finding container 0b15d8e8b7f2a68132d8b2693ca0cac1850d12c0f9e84129afa363a163a2a0a0: Status 404 returned error can't find the container with id 0b15d8e8b7f2a68132d8b2693ca0cac1850d12c0f9e84129afa363a163a2a0a0 Nov 01 20:42:02.423253 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:02.423216 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" event={"ID":"0347f23c-9e24-472c-b4e0-0da2661b8f6c","Type":"ContainerStarted","Data":"0b15d8e8b7f2a68132d8b2693ca0cac1850d12c0f9e84129afa363a163a2a0a0"} Nov 01 20:42:03.430580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:03.430541 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" event={"ID":"0347f23c-9e24-472c-b4e0-0da2661b8f6c","Type":"ContainerStarted","Data":"b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048"} Nov 01 20:42:03.431054 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:03.430664 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:03.546734 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:03.546675 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" podStartSLOduration=2.761568135 podStartE2EDuration="3.546658327s" totalImagesPullingTime="785.090192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:42:03.544160335 +0000 UTC m=+677.194211903" watchObservedRunningTime="2026-11-01 20:42:03.546658327 +0000 UTC m=+677.196709872" Nov 01 20:42:06.036219 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:06.036185 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:42:06.424692 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:06.423926 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 20:42:06.563331 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:42:06.563285 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:42:09.441386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:09.441354 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 20:42:23.172395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:23.172358 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:42:36.553182 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:42:36.553134 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:42:37.580548 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:37.580511 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:42:50.459800 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.459763 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc"] Nov 01 20:42:50.464039 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.464019 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.562411 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.562374 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-tmp-dir\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.562638 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.562421 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rbm\" (UniqueName: \"kubernetes.io/projected/00b8771f-982e-4022-a1de-ab6373b7dab1-kube-api-access-82rbm\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.562638 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.562609 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-dshm\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.562771 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.562645 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-model-cache\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.562771 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.562670 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b8771f-982e-4022-a1de-ab6373b7dab1-tls-certs\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.562854 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.562790 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-home\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.566203 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.566181 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"openshift-service-ca.crt\"" Nov 01 20:42:50.566360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.566237 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"kube-root-ca.crt\"" Nov 01 20:42:50.566432 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.566415 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"default-dockercfg-lksmz\"" Nov 01 20:42:50.566670 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.566654 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"simulator-w0-208f2b-kserve-self-signed-certs\"" Nov 01 20:42:50.568796 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.568775 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc"] Nov 01 20:42:50.666433 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.666395 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-tmp-dir\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.666433 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.666446 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rbm\" (UniqueName: \"kubernetes.io/projected/00b8771f-982e-4022-a1de-ab6373b7dab1-kube-api-access-82rbm\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.666735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.666533 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-dshm\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.666735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.666558 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-model-cache\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.666735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.666580 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b8771f-982e-4022-a1de-ab6373b7dab1-tls-certs\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.666735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.666613 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-home\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.666988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.666963 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-tmp-dir\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.667045 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.667003 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-home\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.667215 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.667190 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-model-cache\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.669296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.669274 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b8771f-982e-4022-a1de-ab6373b7dab1-tls-certs\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.669972 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.669950 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-dshm\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.724614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.724533 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rbm\" (UniqueName: \"kubernetes.io/projected/00b8771f-982e-4022-a1de-ab6373b7dab1-kube-api-access-82rbm\") pod \"simulator-w0-208f2b-kserve-7bb675d57f-wt5fc\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:50.774736 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:50.774700 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:51.153920 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:51.153896 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc"] Nov 01 20:42:51.156313 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:42:51.156285 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00b8771f_982e_4022_a1de_ab6373b7dab1.slice/crio-b7dbeb3de658e19324383e9a114891f4518d70a8f7a293fe57eb817685a4c0f6 WatchSource:0}: Error finding container b7dbeb3de658e19324383e9a114891f4518d70a8f7a293fe57eb817685a4c0f6: Status 404 returned error can't find the container with id b7dbeb3de658e19324383e9a114891f4518d70a8f7a293fe57eb817685a4c0f6 Nov 01 20:42:51.701701 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:51.701662 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" event={"ID":"00b8771f-982e-4022-a1de-ab6373b7dab1","Type":"ContainerStarted","Data":"b7dbeb3de658e19324383e9a114891f4518d70a8f7a293fe57eb817685a4c0f6"} Nov 01 20:42:53.713525 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:53.713421 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" event={"ID":"00b8771f-982e-4022-a1de-ab6373b7dab1","Type":"ContainerStarted","Data":"9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4"} Nov 01 20:42:53.775005 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:53.774970 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:42:53.827635 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:53.827574 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podStartSLOduration=1.579104837 podStartE2EDuration="3.827558306s" totalImagesPullingTime="2.248453469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:42:53.826118869 +0000 UTC m=+727.476170433" watchObservedRunningTime="2026-11-01 20:42:53.827558306 +0000 UTC m=+727.477609884" Nov 01 20:42:57.244618 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:42:57.244586 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:43:00.775795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:00.775761 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:43:00.788578 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:00.788546 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:43:01.533132 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.533097 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m"] Nov 01 20:43:01.535626 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.535601 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.584721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.584685 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tmp-dir\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.584721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.584721 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tls-certs\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.584942 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.584745 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2rm\" (UniqueName: \"kubernetes.io/projected/0288ca11-f321-4f10-b59f-8f1a4e7c293b-kube-api-access-6v2rm\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.584942 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.584867 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-model-cache\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.584942 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.584908 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-dshm\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.585117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.585067 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-home\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.587871 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.587846 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"default-dockercfg-ck7jd\"" Nov 01 20:43:01.587871 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.587846 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"openshift-service-ca.crt\"" Nov 01 20:43:01.621952 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.621926 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"simulator-w2-cd248f-kserve-self-signed-certs\"" Nov 01 20:43:01.640090 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.640040 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m"] Nov 01 20:43:01.685614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.685583 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-home\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.685614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.685620 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tmp-dir\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.685875 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.685639 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tls-certs\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.685875 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.685661 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2rm\" (UniqueName: \"kubernetes.io/projected/0288ca11-f321-4f10-b59f-8f1a4e7c293b-kube-api-access-6v2rm\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.685875 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.685680 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-model-cache\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.685875 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.685807 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-dshm\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.686103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.686023 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-home\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.686151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.686099 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tmp-dir\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.686151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.686140 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-model-cache\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.687949 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.687929 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"kube-root-ca.crt\"" Nov 01 20:43:01.688258 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.688236 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-dshm\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.688443 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.688429 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tls-certs\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.745484 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.745457 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2rm\" (UniqueName: \"kubernetes.io/projected/0288ca11-f321-4f10-b59f-8f1a4e7c293b-kube-api-access-6v2rm\") pod \"simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:01.771692 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.771663 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:43:01.847416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:01.847371 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:02.246441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:02.246411 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m"] Nov 01 20:43:02.249025 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:43:02.248993 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0288ca11_f321_4f10_b59f_8f1a4e7c293b.slice/crio-adbecb667cd1c56a7fb3376b2adab4637c5638d3d57e363b50ac8d67184262c3 WatchSource:0}: Error finding container adbecb667cd1c56a7fb3376b2adab4637c5638d3d57e363b50ac8d67184262c3: Status 404 returned error can't find the container with id adbecb667cd1c56a7fb3376b2adab4637c5638d3d57e363b50ac8d67184262c3 Nov 01 20:43:02.766322 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:02.766281 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" event={"ID":"0288ca11-f321-4f10-b59f-8f1a4e7c293b","Type":"ContainerStarted","Data":"13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8"} Nov 01 20:43:02.766568 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:02.766335 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" event={"ID":"0288ca11-f321-4f10-b59f-8f1a4e7c293b","Type":"ContainerStarted","Data":"adbecb667cd1c56a7fb3376b2adab4637c5638d3d57e363b50ac8d67184262c3"} Nov 01 20:43:02.847833 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:02.847803 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:02.879997 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:02.879940 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podStartSLOduration=1.8799250939999999 podStartE2EDuration="1.879925094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:02.879047464 +0000 UTC m=+736.529099022" watchObservedRunningTime="2026-11-01 20:43:02.879925094 +0000 UTC m=+736.529976639" Nov 01 20:43:06.559640 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:43:06.559600 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:43:11.847663 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:11.847627 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:11.860121 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:11.860096 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:12.833691 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:12.833660 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:43:23.036159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:23.036120 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:43:35.651466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.651371 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg"] Nov 01 20:43:35.655341 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.655319 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.707293 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.707271 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-944263-openshift-default-dockercfg-s5hxs\"" Nov 01 20:43:35.732895 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.732869 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxjx\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-kube-api-access-hnxjx\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.733024 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.732932 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.733088 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.733051 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.733173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.733151 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.733238 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.733221 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.733318 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.733304 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.733360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.733336 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.733395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.733362 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.733462 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.733446 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.759316 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.759281 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg"] Nov 01 20:43:35.835297 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835265 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.835590 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835561 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.835725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835602 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.835725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835627 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.835725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835657 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.835725 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835690 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxjx\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-kube-api-access-hnxjx\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.836039 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835788 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.836039 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835823 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.836039 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.835851 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.836039 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.836016 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.836364 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.836166 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.837003 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.836979 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.837322 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.837299 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.837574 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.837556 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.840004 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.839978 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.840774 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.840753 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.892512 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.892476 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.893262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.893238 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxjx\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-kube-api-access-hnxjx\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:35.971576 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:35.971472 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:36.432605 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:36.432559 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg"] Nov 01 20:43:36.434899 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:43:36.434867 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f79c420_64c0_464f_b08f_ceeef1b31aa5.slice/crio-19ccffa7604e3a4cd7b97b7a9ee38eb8000e65419ee524d595a89827a47c3a8f WatchSource:0}: Error finding container 19ccffa7604e3a4cd7b97b7a9ee38eb8000e65419ee524d595a89827a47c3a8f: Status 404 returned error can't find the container with id 19ccffa7604e3a4cd7b97b7a9ee38eb8000e65419ee524d595a89827a47c3a8f Nov 01 20:43:36.437498 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:36.437458 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:43:36.437598 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:36.437547 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:43:36.437658 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:36.437593 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:43:36.558305 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:43:36.558268 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:43:36.976680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:36.976637 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" event={"ID":"3f79c420-64c0-464f-b08f-ceeef1b31aa5","Type":"ContainerStarted","Data":"01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199"} Nov 01 20:43:36.977111 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:36.976707 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" event={"ID":"3f79c420-64c0-464f-b08f-ceeef1b31aa5","Type":"ContainerStarted","Data":"19ccffa7604e3a4cd7b97b7a9ee38eb8000e65419ee524d595a89827a47c3a8f"} Nov 01 20:43:37.097914 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:37.097839 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" podStartSLOduration=2.097819645 podStartE2EDuration="2.097819645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:37.097645285 +0000 UTC m=+770.747696854" watchObservedRunningTime="2026-11-01 20:43:37.097819645 +0000 UTC m=+770.747871190" Nov 01 20:43:37.972279 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:37.972227 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:37.973627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:37.973600 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:43:37.973742 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:37.973651 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:43:38.972171 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:38.972136 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:43:38.972667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:38.972197 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:43:39.972252 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:39.972222 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:43:39.972648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:39.972274 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:43:40.972937 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:40.972898 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:43:40.973398 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:40.972950 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:43:41.971990 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:41.971947 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:43:41.972245 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:41.972011 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:43:43.160934 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:43.160895 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:43.161677 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:43.161649 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:43.162665 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:43.162635 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 20:43:44.928426 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:44.928388 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp"] Nov 01 20:43:44.932907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:44.932882 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:44.989649 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:44.989618 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp"] Nov 01 20:43:45.033980 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.033938 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"kube-root-ca.crt\"" Nov 01 20:43:45.033980 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.033944 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"openshift-service-ca.crt\"" Nov 01 20:43:45.036498 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.036474 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-model-cache\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.036606 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.036514 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-dshm\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.036606 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.036545 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-home\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.036713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.036617 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"default-dockercfg-dssl5\"" Nov 01 20:43:45.036713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.036618 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-tmp-dir\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.036713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.036690 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"premium-w1-db0157-kserve-self-signed-certs\"" Nov 01 20:43:45.036863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.036703 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rsnz\" (UniqueName: \"kubernetes.io/projected/d6ec5583-c611-4b43-9233-b5bd641d801d-kube-api-access-4rsnz\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.036863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.036819 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ec5583-c611-4b43-9233-b5bd641d801d-tls-certs\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.137518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.137482 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ec5583-c611-4b43-9233-b5bd641d801d-tls-certs\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.137741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.137633 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-model-cache\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.137741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.137661 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-dshm\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.137741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.137691 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-home\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.137741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.137715 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-tmp-dir\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.137966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.137762 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsnz\" (UniqueName: \"kubernetes.io/projected/d6ec5583-c611-4b43-9233-b5bd641d801d-kube-api-access-4rsnz\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.138185 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.138160 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-tmp-dir\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.138503 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.138478 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-model-cache\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.138897 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.138874 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-home\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.140442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.140421 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-dshm\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.141003 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.140983 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ec5583-c611-4b43-9233-b5bd641d801d-tls-certs\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.194292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.194212 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rsnz\" (UniqueName: \"kubernetes.io/projected/d6ec5583-c611-4b43-9233-b5bd641d801d-kube-api-access-4rsnz\") pod \"premium-w1-db0157-kserve-75fb4d65cd-r7zcp\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.246348 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.246292 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:45.660954 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:45.660925 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp"] Nov 01 20:43:45.663137 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:43:45.663109 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ec5583_c611_4b43_9233_b5bd641d801d.slice/crio-a0d274040972de8f76d399db170cf87cef5b04e996472444115b48d5f9dc87b7 WatchSource:0}: Error finding container a0d274040972de8f76d399db170cf87cef5b04e996472444115b48d5f9dc87b7: Status 404 returned error can't find the container with id a0d274040972de8f76d399db170cf87cef5b04e996472444115b48d5f9dc87b7 Nov 01 20:43:46.030586 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:46.030492 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" event={"ID":"d6ec5583-c611-4b43-9233-b5bd641d801d","Type":"ContainerStarted","Data":"095fd399398915640c9bac96ee5792104f369b3274e50164a7f713122ea2c8c5"} Nov 01 20:43:46.030586 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:46.030550 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" event={"ID":"d6ec5583-c611-4b43-9233-b5bd641d801d","Type":"ContainerStarted","Data":"a0d274040972de8f76d399db170cf87cef5b04e996472444115b48d5f9dc87b7"} Nov 01 20:43:46.147667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:46.147613 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podStartSLOduration=2.147598451 podStartE2EDuration="2.147598451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:46.144951744 +0000 UTC m=+779.795003313" watchObservedRunningTime="2026-11-01 20:43:46.147598451 +0000 UTC m=+779.797649996" Nov 01 20:43:46.246668 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:46.246636 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:46.276410 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:46.276359 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:43:55.246712 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:55.246676 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:55.259527 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:55.259500 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:43:56.099335 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:43:56.099307 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:44:06.555831 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:44:06.555793 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:44:09.234323 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:09.234286 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:44:21.930245 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:21.930206 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2"] Nov 01 20:44:21.935503 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:21.935482 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:21.987557 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:21.987521 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-db0157-openshift-default-dockercfg-gx5m8\"" Nov 01 20:44:22.022260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022225 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2wg2\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-kube-api-access-x2wg2\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.022260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022261 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.022492 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022283 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.022492 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022329 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.022492 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022396 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e03d930-b982-4d8d-a611-31fdb427e38e-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.022492 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022422 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.022492 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022447 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.022492 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022464 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.022741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.022626 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.040486 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.040464 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2"] Nov 01 20:44:22.123291 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123265 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123342 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2wg2\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-kube-api-access-x2wg2\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123361 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123381 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123401 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123428 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e03d930-b982-4d8d-a611-31fdb427e38e-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123719 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123498 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123719 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123536 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123719 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123560 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123868 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123847 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123906 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123885 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.123948 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.123935 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.124196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.124182 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.124415 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.124396 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e03d930-b982-4d8d-a611-31fdb427e38e-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.125903 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.125884 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.126166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.126148 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.179833 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.179810 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.179967 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.179948 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2wg2\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-kube-api-access-x2wg2\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.248688 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.248625 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:22.675094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.675044 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2"] Nov 01 20:44:22.676474 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:44:22.676449 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e03d930_b982_4d8d_a611_31fdb427e38e.slice/crio-3a0ca72645d432935650d6084e8d996a2edbe4cd7bdeb29ec82c1d7aae51b13d WatchSource:0}: Error finding container 3a0ca72645d432935650d6084e8d996a2edbe4cd7bdeb29ec82c1d7aae51b13d: Status 404 returned error can't find the container with id 3a0ca72645d432935650d6084e8d996a2edbe4cd7bdeb29ec82c1d7aae51b13d Nov 01 20:44:22.678805 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.678773 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:44:22.678891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.678835 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:44:22.678891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:22.678865 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:44:23.249750 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:23.249710 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" event={"ID":"3e03d930-b982-4d8d-a611-31fdb427e38e","Type":"ContainerStarted","Data":"12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b"} Nov 01 20:44:23.250210 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:23.249766 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" event={"ID":"3e03d930-b982-4d8d-a611-31fdb427e38e","Type":"ContainerStarted","Data":"3a0ca72645d432935650d6084e8d996a2edbe4cd7bdeb29ec82c1d7aae51b13d"} Nov 01 20:44:23.367300 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:23.367245 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" podStartSLOduration=2.367228739 podStartE2EDuration="2.367228739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:23.364250358 +0000 UTC m=+817.014301939" watchObservedRunningTime="2026-11-01 20:44:23.367228739 +0000 UTC m=+817.017280283" Nov 01 20:44:24.249766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:24.249730 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:24.250957 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:24.250935 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 20:44:24.251050 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:24.250987 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 20:44:25.249269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:25.249229 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 20:44:25.249461 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:25.249309 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 20:44:26.249673 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:26.249637 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 20:44:26.250303 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:26.250269 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 20:44:27.249592 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:27.249559 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 20:44:27.249773 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:27.249614 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 20:44:28.249781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:28.249748 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 20:44:28.250199 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:28.249807 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 20:44:29.431004 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:29.430971 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:29.431924 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:29.431900 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:29.433028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:29.433009 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 20:44:36.556271 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:44:36.556106 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:44:37.363918 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.363889 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj"] Nov 01 20:44:37.367486 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.367470 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.420693 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.420666 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-208f2b-openshift-default-dockercfg-x8lc4\"" Nov 01 20:44:37.473228 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.473195 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj"] Nov 01 20:44:37.493377 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493340 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.493579 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493405 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.493579 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493469 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926f1d5c-a975-439a-b8b3-b642316d7882-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.493579 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493541 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926f1d5c-a975-439a-b8b3-b642316d7882-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.493724 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493664 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.493724 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493703 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.493802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493767 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.493848 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493800 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.493848 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.493840 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md8m7\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-kube-api-access-md8m7\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.594458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594415 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.594458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594457 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926f1d5c-a975-439a-b8b3-b642316d7882-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594489 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926f1d5c-a975-439a-b8b3-b642316d7882-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594559 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594604 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594675 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594741 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594769 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md8m7\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-kube-api-access-md8m7\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594850 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.594907 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595465 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.595211 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595465 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.595263 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595465 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.595352 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926f1d5c-a975-439a-b8b3-b642316d7882-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.595465 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.595419 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.597551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.597530 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.597780 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.597762 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926f1d5c-a975-439a-b8b3-b642316d7882-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.651322 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.651247 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.651458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.651358 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md8m7\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-kube-api-access-md8m7\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:37.681265 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:37.681230 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:38.116914 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:38.116883 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj"] Nov 01 20:44:38.117710 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:44:38.117683 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926f1d5c_a975_439a_b8b3_b642316d7882.slice/crio-8544e44b7136b344295b4164ec664a8abf013c0aea42e16dd8659f508c77dad2 WatchSource:0}: Error finding container 8544e44b7136b344295b4164ec664a8abf013c0aea42e16dd8659f508c77dad2: Status 404 returned error can't find the container with id 8544e44b7136b344295b4164ec664a8abf013c0aea42e16dd8659f508c77dad2 Nov 01 20:44:38.120265 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:38.120224 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:44:38.120353 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:38.120315 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:44:38.120393 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:38.120360 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:44:38.344903 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:38.344863 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" event={"ID":"926f1d5c-a975-439a-b8b3-b642316d7882","Type":"ContainerStarted","Data":"52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b"} Nov 01 20:44:38.345124 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:38.344931 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" event={"ID":"926f1d5c-a975-439a-b8b3-b642316d7882","Type":"ContainerStarted","Data":"8544e44b7136b344295b4164ec664a8abf013c0aea42e16dd8659f508c77dad2"} Nov 01 20:44:38.460555 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:38.460427 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" podStartSLOduration=1.460410451 podStartE2EDuration="1.460410451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:38.459693824 +0000 UTC m=+832.109745395" watchObservedRunningTime="2026-11-01 20:44:38.460410451 +0000 UTC m=+832.110461998" Nov 01 20:44:38.681555 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:38.681524 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:39.683970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:39.683934 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:44:39.684406 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:39.684004 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:44:40.682314 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:40.682278 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:44:40.682600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:40.682356 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:44:41.681790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:41.681750 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:44:41.682244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:41.681813 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:44:42.681724 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:42.681692 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:44:42.681901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:42.681749 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:44:43.681668 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:43.681629 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:44:43.681964 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:43.681931 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:44:44.245450 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:44.245416 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:44:44.938113 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:44.937572 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:44.939734 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:44.939702 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:44:44.941772 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:44:44.941743 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 20:45:00.166096 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.165999 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d"] Nov 01 20:45:00.171518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.171484 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc"] Nov 01 20:45:00.171663 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.171628 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:00.174966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.174947 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:00.187780 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.187748 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv"] Nov 01 20:45:00.191761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.191740 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:00.225183 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.225160 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:45:00.258371 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.258327 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc\" (UID: \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:00.258539 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.258470 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa9973-d0bc-48cd-a137-05a036138f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv\" (UID: \"a9fa9973-d0bc-48cd-a137-05a036138f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:00.258611 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.258546 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv79h\" (UniqueName: \"kubernetes.io/projected/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-kube-api-access-jv79h\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc\" (UID: \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:00.258689 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.258671 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459l5\" (UniqueName: \"kubernetes.io/projected/2064a78b-5470-4026-991e-de1013ea2cce-kube-api-access-459l5\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d\" (UID: \"2064a78b-5470-4026-991e-de1013ea2cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:00.258733 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.258710 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2064a78b-5470-4026-991e-de1013ea2cce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d\" (UID: \"2064a78b-5470-4026-991e-de1013ea2cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:00.258785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.258733 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjplr\" (UniqueName: \"kubernetes.io/projected/a9fa9973-d0bc-48cd-a137-05a036138f13-kube-api-access-tjplr\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv\" (UID: \"a9fa9973-d0bc-48cd-a137-05a036138f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:00.277989 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.275729 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d"] Nov 01 20:45:00.277989 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.277897 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc"] Nov 01 20:45:00.331800 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.324903 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv"] Nov 01 20:45:00.359592 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.359563 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc\" (UID: \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:00.359799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.359616 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa9973-d0bc-48cd-a137-05a036138f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv\" (UID: \"a9fa9973-d0bc-48cd-a137-05a036138f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:00.359799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.359644 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv79h\" (UniqueName: \"kubernetes.io/projected/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-kube-api-access-jv79h\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc\" (UID: \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:00.359799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.359691 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459l5\" (UniqueName: \"kubernetes.io/projected/2064a78b-5470-4026-991e-de1013ea2cce-kube-api-access-459l5\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d\" (UID: \"2064a78b-5470-4026-991e-de1013ea2cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:00.359799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.359730 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2064a78b-5470-4026-991e-de1013ea2cce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d\" (UID: \"2064a78b-5470-4026-991e-de1013ea2cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:00.359799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.359750 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjplr\" (UniqueName: \"kubernetes.io/projected/a9fa9973-d0bc-48cd-a137-05a036138f13-kube-api-access-tjplr\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv\" (UID: \"a9fa9973-d0bc-48cd-a137-05a036138f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:00.360487 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.360459 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa9973-d0bc-48cd-a137-05a036138f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv\" (UID: \"a9fa9973-d0bc-48cd-a137-05a036138f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:00.360609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.360459 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc\" (UID: \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:00.360609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.360499 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2064a78b-5470-4026-991e-de1013ea2cce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d\" (UID: \"2064a78b-5470-4026-991e-de1013ea2cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:00.416707 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.416621 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-459l5\" (UniqueName: \"kubernetes.io/projected/2064a78b-5470-4026-991e-de1013ea2cce-kube-api-access-459l5\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d\" (UID: \"2064a78b-5470-4026-991e-de1013ea2cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:00.416881 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.416858 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv79h\" (UniqueName: \"kubernetes.io/projected/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-kube-api-access-jv79h\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc\" (UID: \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:00.417290 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.417269 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjplr\" (UniqueName: \"kubernetes.io/projected/a9fa9973-d0bc-48cd-a137-05a036138f13-kube-api-access-tjplr\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv\" (UID: \"a9fa9973-d0bc-48cd-a137-05a036138f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:00.513480 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.513444 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:00.521262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.521226 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:00.531277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.531258 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:00.927043 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.927010 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d"] Nov 01 20:45:00.927993 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:45:00.927962 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2064a78b_5470_4026_991e_de1013ea2cce.slice/crio-10dbb5094aff1c613e9a38346526c960ab104aae5bf5d0cd194d2d69cb0e0d6b WatchSource:0}: Error finding container 10dbb5094aff1c613e9a38346526c960ab104aae5bf5d0cd194d2d69cb0e0d6b: Status 404 returned error can't find the container with id 10dbb5094aff1c613e9a38346526c960ab104aae5bf5d0cd194d2d69cb0e0d6b Nov 01 20:45:00.978864 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.978784 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv"] Nov 01 20:45:00.983264 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:00.982761 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc"] Nov 01 20:45:00.983264 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:45:00.983187 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fa9973_d0bc_48cd_a137_05a036138f13.slice/crio-f817a8bae7351078ee1efaf094e5788e84d902563828970d2fa126236fa25ce5 WatchSource:0}: Error finding container f817a8bae7351078ee1efaf094e5788e84d902563828970d2fa126236fa25ce5: Status 404 returned error can't find the container with id f817a8bae7351078ee1efaf094e5788e84d902563828970d2fa126236fa25ce5 Nov 01 20:45:00.983578 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:45:00.983547 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa2aeb7_e831_4fc6_a76e_7ce52ad53106.slice/crio-ef0064ecc064c6f62ffa1973eb8b38ecd97cb8f64ca1324d1b332829757dcbb0 WatchSource:0}: Error finding container ef0064ecc064c6f62ffa1973eb8b38ecd97cb8f64ca1324d1b332829757dcbb0: Status 404 returned error can't find the container with id ef0064ecc064c6f62ffa1973eb8b38ecd97cb8f64ca1324d1b332829757dcbb0 Nov 01 20:45:01.521676 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:01.521635 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" event={"ID":"faa2aeb7-e831-4fc6-a76e-7ce52ad53106","Type":"ContainerStarted","Data":"ef0064ecc064c6f62ffa1973eb8b38ecd97cb8f64ca1324d1b332829757dcbb0"} Nov 01 20:45:01.522996 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:01.522968 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" event={"ID":"a9fa9973-d0bc-48cd-a137-05a036138f13","Type":"ContainerStarted","Data":"f817a8bae7351078ee1efaf094e5788e84d902563828970d2fa126236fa25ce5"} Nov 01 20:45:01.524165 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:01.524145 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" event={"ID":"2064a78b-5470-4026-991e-de1013ea2cce","Type":"ContainerStarted","Data":"10dbb5094aff1c613e9a38346526c960ab104aae5bf5d0cd194d2d69cb0e0d6b"} Nov 01 20:45:05.551101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:05.550996 4477 generic.go:356] "Generic (PLEG): container finished" podID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerID="1ffc4f3913a0d7b97bc566bee78f10569cd50e325abbb69ca5c4f0c76c78df9f" exitCode=60 Nov 01 20:45:05.551484 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:05.551104 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" event={"ID":"faa2aeb7-e831-4fc6-a76e-7ce52ad53106","Type":"ContainerDied","Data":"1ffc4f3913a0d7b97bc566bee78f10569cd50e325abbb69ca5c4f0c76c78df9f"} Nov 01 20:45:05.551484 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:05.551375 4477 scope.go:122] "RemoveContainer" containerID="1ffc4f3913a0d7b97bc566bee78f10569cd50e325abbb69ca5c4f0c76c78df9f" Nov 01 20:45:06.554579 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:06.554468 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:45:06.559517 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.559487 4477 generic.go:356] "Generic (PLEG): container finished" podID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerID="1044cb7262dfb209a381f19d23e12e45b72d59e265369c9927bf325379b39237" exitCode=60 Nov 01 20:45:06.559682 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.559596 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" event={"ID":"faa2aeb7-e831-4fc6-a76e-7ce52ad53106","Type":"ContainerDied","Data":"1044cb7262dfb209a381f19d23e12e45b72d59e265369c9927bf325379b39237"} Nov 01 20:45:06.559755 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.559657 4477 scope.go:122] "RemoveContainer" containerID="1ffc4f3913a0d7b97bc566bee78f10569cd50e325abbb69ca5c4f0c76c78df9f" Nov 01 20:45:06.559936 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.559918 4477 scope.go:122] "RemoveContainer" containerID="1044cb7262dfb209a381f19d23e12e45b72d59e265369c9927bf325379b39237" Nov 01 20:45:06.560223 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:06.560200 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc_odh-ai-gateway-infra(faa2aeb7-e831-4fc6-a76e-7ce52ad53106)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" Nov 01 20:45:06.562026 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.561551 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" event={"ID":"a9fa9973-d0bc-48cd-a137-05a036138f13","Type":"ContainerStarted","Data":"f96a6d697693419ace612d028e3447b53862c543b1a561b88b95b5f5c5d468cb"} Nov 01 20:45:06.565589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.565561 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" event={"ID":"2064a78b-5470-4026-991e-de1013ea2cce","Type":"ContainerStarted","Data":"db8154e1056bf3ca2da39083cd018bd80d1492367d9de1651f9d886f1def4a95"} Nov 01 20:45:06.648213 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.648177 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5"] Nov 01 20:45:06.653816 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.653794 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.706299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.706267 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cd248f-openshift-default-dockercfg-6v4rd\"" Nov 01 20:45:06.708898 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.708868 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5"] Nov 01 20:45:06.731512 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731482 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.731648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731524 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.731648 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731594 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.731760 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731670 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.731760 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731731 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.731866 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731783 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.731866 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731825 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwbhz\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-kube-api-access-jwbhz\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.731866 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731852 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.732007 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.731977 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.787924 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.787870 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" podStartSLOduration=1.935208244 podStartE2EDuration="6.787853372s" totalImagesPullingTime="4.852645128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:06.785562189 +0000 UTC m=+860.435613735" watchObservedRunningTime="2026-11-01 20:45:06.787853372 +0000 UTC m=+860.437904958" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.832698 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbhz\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-kube-api-access-jwbhz\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.832754 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.832830 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.832880 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.832914 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.832966 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.832998 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.833048 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833095 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.833103 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.833413 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.833711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.833663 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.834379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.833858 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.834379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.833880 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.834379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.834051 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.836319 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.836288 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.837146 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.837119 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.889833 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.889793 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.889993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.889905 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbhz\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-kube-api-access-jwbhz\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:06.969502 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:06.969467 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:07.011698 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.011372 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" podStartSLOduration=1.903062262 podStartE2EDuration="7.011352344s" totalImagesPullingTime="5.108290082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:07.009737084 +0000 UTC m=+860.659788629" watchObservedRunningTime="2026-11-01 20:45:07.011352344 +0000 UTC m=+860.661403890" Nov 01 20:45:07.407833 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.407804 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5"] Nov 01 20:45:07.409298 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:45:07.409270 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06ec19e_e8fd_4b6e_879e_e5e3092a0315.slice/crio-7b9dbdeda0f1cd8283b336f9dee9aec7ef21e857c58c3a9552f9b5c28682ab8a WatchSource:0}: Error finding container 7b9dbdeda0f1cd8283b336f9dee9aec7ef21e857c58c3a9552f9b5c28682ab8a: Status 404 returned error can't find the container with id 7b9dbdeda0f1cd8283b336f9dee9aec7ef21e857c58c3a9552f9b5c28682ab8a Nov 01 20:45:07.411625 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.411593 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:45:07.411704 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.411656 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:45:07.411743 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.411703 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:45:07.572988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.572951 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" event={"ID":"e06ec19e-e8fd-4b6e-879e-e5e3092a0315","Type":"ContainerStarted","Data":"5bec88da5b9177e025ebd399ceed60c0148fe35dc1eb57ea0c57271331289be8"} Nov 01 20:45:07.573493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.573009 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" event={"ID":"e06ec19e-e8fd-4b6e-879e-e5e3092a0315","Type":"ContainerStarted","Data":"7b9dbdeda0f1cd8283b336f9dee9aec7ef21e857c58c3a9552f9b5c28682ab8a"} Nov 01 20:45:07.575401 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.575380 4477 scope.go:122] "RemoveContainer" containerID="1044cb7262dfb209a381f19d23e12e45b72d59e265369c9927bf325379b39237" Nov 01 20:45:07.575703 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:07.575673 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc_odh-ai-gateway-infra(faa2aeb7-e831-4fc6-a76e-7ce52ad53106)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" Nov 01 20:45:07.577028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.577002 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerID="f96a6d697693419ace612d028e3447b53862c543b1a561b88b95b5f5c5d468cb" exitCode=60 Nov 01 20:45:07.577141 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.577061 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" event={"ID":"a9fa9973-d0bc-48cd-a137-05a036138f13","Type":"ContainerDied","Data":"f96a6d697693419ace612d028e3447b53862c543b1a561b88b95b5f5c5d468cb"} Nov 01 20:45:07.577293 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.577277 4477 scope.go:122] "RemoveContainer" containerID="f96a6d697693419ace612d028e3447b53862c543b1a561b88b95b5f5c5d468cb" Nov 01 20:45:07.578615 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.578596 4477 generic.go:356] "Generic (PLEG): container finished" podID="2064a78b-5470-4026-991e-de1013ea2cce" containerID="db8154e1056bf3ca2da39083cd018bd80d1492367d9de1651f9d886f1def4a95" exitCode=60 Nov 01 20:45:07.578712 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.578667 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" event={"ID":"2064a78b-5470-4026-991e-de1013ea2cce","Type":"ContainerDied","Data":"db8154e1056bf3ca2da39083cd018bd80d1492367d9de1651f9d886f1def4a95"} Nov 01 20:45:07.578839 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.578823 4477 scope.go:122] "RemoveContainer" containerID="db8154e1056bf3ca2da39083cd018bd80d1492367d9de1651f9d886f1def4a95" Nov 01 20:45:07.693153 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.692416 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" podStartSLOduration=1.6923952450000002 podStartE2EDuration="1.692395245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:07.691575169 +0000 UTC m=+861.341626732" watchObservedRunningTime="2026-11-01 20:45:07.692395245 +0000 UTC m=+861.342446793" Nov 01 20:45:07.969703 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:07.969605 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:08.587265 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.587233 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerID="ba15687cc3c3d6ff8ad3d31d865362dfcd9f5b3c4d8ce4e85394bf9e178b217f" exitCode=60 Nov 01 20:45:08.587660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.587300 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" event={"ID":"a9fa9973-d0bc-48cd-a137-05a036138f13","Type":"ContainerDied","Data":"ba15687cc3c3d6ff8ad3d31d865362dfcd9f5b3c4d8ce4e85394bf9e178b217f"} Nov 01 20:45:08.587660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.587359 4477 scope.go:122] "RemoveContainer" containerID="f96a6d697693419ace612d028e3447b53862c543b1a561b88b95b5f5c5d468cb" Nov 01 20:45:08.587660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.587550 4477 scope.go:122] "RemoveContainer" containerID="ba15687cc3c3d6ff8ad3d31d865362dfcd9f5b3c4d8ce4e85394bf9e178b217f" Nov 01 20:45:08.587834 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:08.587816 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv_odh-ai-gateway-infra(a9fa9973-d0bc-48cd-a137-05a036138f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" Nov 01 20:45:08.589378 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.589357 4477 generic.go:356] "Generic (PLEG): container finished" podID="2064a78b-5470-4026-991e-de1013ea2cce" containerID="f83c603c1cc94604efea6f0ed9239b5c7ce4caa9c7ae85cf987beb0c7adcf616" exitCode=60 Nov 01 20:45:08.589497 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.589428 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" event={"ID":"2064a78b-5470-4026-991e-de1013ea2cce","Type":"ContainerDied","Data":"f83c603c1cc94604efea6f0ed9239b5c7ce4caa9c7ae85cf987beb0c7adcf616"} Nov 01 20:45:08.589663 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.589649 4477 scope.go:122] "RemoveContainer" containerID="f83c603c1cc94604efea6f0ed9239b5c7ce4caa9c7ae85cf987beb0c7adcf616" Nov 01 20:45:08.589900 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:08.589883 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d_odh-ai-gateway-infra(2064a78b-5470-4026-991e-de1013ea2cce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" podUID="2064a78b-5470-4026-991e-de1013ea2cce" Nov 01 20:45:08.599864 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.599847 4477 scope.go:122] "RemoveContainer" containerID="db8154e1056bf3ca2da39083cd018bd80d1492367d9de1651f9d886f1def4a95" Nov 01 20:45:08.972401 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.972367 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:45:08.972572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:08.972420 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:45:09.597397 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:09.597364 4477 scope.go:122] "RemoveContainer" containerID="ba15687cc3c3d6ff8ad3d31d865362dfcd9f5b3c4d8ce4e85394bf9e178b217f" Nov 01 20:45:09.597872 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:09.597628 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv_odh-ai-gateway-infra(a9fa9973-d0bc-48cd-a137-05a036138f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" Nov 01 20:45:09.599002 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:09.598980 4477 scope.go:122] "RemoveContainer" containerID="f83c603c1cc94604efea6f0ed9239b5c7ce4caa9c7ae85cf987beb0c7adcf616" Nov 01 20:45:09.599277 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:09.599254 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d_odh-ai-gateway-infra(2064a78b-5470-4026-991e-de1013ea2cce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" podUID="2064a78b-5470-4026-991e-de1013ea2cce" Nov 01 20:45:09.970950 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:09.970865 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:45:09.970950 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:09.970933 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:45:10.969970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:10.969928 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:45:10.970388 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:10.970008 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:45:11.970708 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:11.970665 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:45:11.971242 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:11.970721 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:45:12.970964 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:12.970927 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:45:12.971434 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:12.970989 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:45:14.102371 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:14.102308 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:14.102853 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:14.102718 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:14.103779 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:14.103749 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 20:45:19.497053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:19.497024 4477 scope.go:122] "RemoveContainer" containerID="1044cb7262dfb209a381f19d23e12e45b72d59e265369c9927bf325379b39237" Nov 01 20:45:20.691762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:20.691669 4477 generic.go:356] "Generic (PLEG): container finished" podID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerID="fc26ce3c72c66a1265f0ac8aede6f16175f91df8437d94b8f6915f2f614e8f6b" exitCode=60 Nov 01 20:45:20.692162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:20.691754 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" event={"ID":"faa2aeb7-e831-4fc6-a76e-7ce52ad53106","Type":"ContainerDied","Data":"fc26ce3c72c66a1265f0ac8aede6f16175f91df8437d94b8f6915f2f614e8f6b"} Nov 01 20:45:20.692162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:20.691831 4477 scope.go:122] "RemoveContainer" containerID="1044cb7262dfb209a381f19d23e12e45b72d59e265369c9927bf325379b39237" Nov 01 20:45:20.692296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:20.692272 4477 scope.go:122] "RemoveContainer" containerID="fc26ce3c72c66a1265f0ac8aede6f16175f91df8437d94b8f6915f2f614e8f6b" Nov 01 20:45:20.692549 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:20.692528 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc_odh-ai-gateway-infra(faa2aeb7-e831-4fc6-a76e-7ce52ad53106)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" Nov 01 20:45:21.813319 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:21.813278 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc"] Nov 01 20:45:21.975860 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:21.975825 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:22.098561 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.098470 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-kube-api-access-jv79h\" (UniqueName: \"kubernetes.io/projected/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-kube-api-access-jv79h\") pod \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\" (UID: \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\") " Nov 01 20:45:22.098729 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.098611 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-maas-api-service-ca\") pod \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\" (UID: \"faa2aeb7-e831-4fc6-a76e-7ce52ad53106\") " Nov 01 20:45:22.099025 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.098977 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-maas-api-service-ca" pod "faa2aeb7-e831-4fc6-a76e-7ce52ad53106" (UID: "faa2aeb7-e831-4fc6-a76e-7ce52ad53106"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:22.101121 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.101043 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-kube-api-access-jv79h" pod "faa2aeb7-e831-4fc6-a76e-7ce52ad53106" (UID: "faa2aeb7-e831-4fc6-a76e-7ce52ad53106"). InnerVolumeSpecName "kube-api-access-jv79h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:22.200320 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.200282 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv79h\" (UniqueName: \"kubernetes.io/projected/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-kube-api-access-jv79h\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:22.200320 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.200321 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa2aeb7-e831-4fc6-a76e-7ce52ad53106-maas-api-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:22.497176 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.497067 4477 scope.go:122] "RemoveContainer" containerID="f83c603c1cc94604efea6f0ed9239b5c7ce4caa9c7ae85cf987beb0c7adcf616" Nov 01 20:45:22.708145 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.708107 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" event={"ID":"faa2aeb7-e831-4fc6-a76e-7ce52ad53106","Type":"ContainerDied","Data":"ef0064ecc064c6f62ffa1973eb8b38ecd97cb8f64ca1324d1b332829757dcbb0"} Nov 01 20:45:22.708145 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.708138 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc" Nov 01 20:45:22.708365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.708176 4477 scope.go:122] "RemoveContainer" containerID="fc26ce3c72c66a1265f0ac8aede6f16175f91df8437d94b8f6915f2f614e8f6b" Nov 01 20:45:22.871509 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.871475 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc"] Nov 01 20:45:22.876597 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:22.876571 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892765-jxmjc"] Nov 01 20:45:23.497180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:23.497151 4477 scope.go:122] "RemoveContainer" containerID="ba15687cc3c3d6ff8ad3d31d865362dfcd9f5b3c4d8ce4e85394bf9e178b217f" Nov 01 20:45:23.718110 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:23.718004 4477 generic.go:356] "Generic (PLEG): container finished" podID="2064a78b-5470-4026-991e-de1013ea2cce" containerID="59befdc56c8233214e21a33397aac04d46b3b0dce034e29cd81c1ba01b0c604f" exitCode=60 Nov 01 20:45:23.718110 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:23.718047 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" event={"ID":"2064a78b-5470-4026-991e-de1013ea2cce","Type":"ContainerDied","Data":"59befdc56c8233214e21a33397aac04d46b3b0dce034e29cd81c1ba01b0c604f"} Nov 01 20:45:23.718315 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:23.718126 4477 scope.go:122] "RemoveContainer" containerID="f83c603c1cc94604efea6f0ed9239b5c7ce4caa9c7ae85cf987beb0c7adcf616" Nov 01 20:45:23.718538 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:23.718508 4477 scope.go:122] "RemoveContainer" containerID="59befdc56c8233214e21a33397aac04d46b3b0dce034e29cd81c1ba01b0c604f" Nov 01 20:45:23.718753 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:23.718734 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d_odh-ai-gateway-infra(2064a78b-5470-4026-991e-de1013ea2cce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" podUID="2064a78b-5470-4026-991e-de1013ea2cce" Nov 01 20:45:24.294144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:24.294105 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:45:24.503988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:24.503954 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" path="/var/lib/kubelet/pods/faa2aeb7-e831-4fc6-a76e-7ce52ad53106/volumes" Nov 01 20:45:24.737818 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:24.737784 4477 generic.go:356] "Generic (PLEG): container finished" podID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerID="3ad05d2853c6df9fb942cda659abf6ec007f198400c2a750d30e9d5db2167556" exitCode=60 Nov 01 20:45:24.738144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:24.737831 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" event={"ID":"a9fa9973-d0bc-48cd-a137-05a036138f13","Type":"ContainerDied","Data":"3ad05d2853c6df9fb942cda659abf6ec007f198400c2a750d30e9d5db2167556"} Nov 01 20:45:24.738486 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:24.738380 4477 scope.go:122] "RemoveContainer" containerID="ba15687cc3c3d6ff8ad3d31d865362dfcd9f5b3c4d8ce4e85394bf9e178b217f" Nov 01 20:45:24.743600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:24.743579 4477 scope.go:122] "RemoveContainer" containerID="3ad05d2853c6df9fb942cda659abf6ec007f198400c2a750d30e9d5db2167556" Nov 01 20:45:24.744112 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:24.744070 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv_odh-ai-gateway-infra(a9fa9973-d0bc-48cd-a137-05a036138f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" Nov 01 20:45:24.849488 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:24.849449 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d"] Nov 01 20:45:25.133802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.128388 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:25.255100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.254507 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2064a78b-5470-4026-991e-de1013ea2cce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2064a78b-5470-4026-991e-de1013ea2cce-maas-api-service-ca\") pod \"2064a78b-5470-4026-991e-de1013ea2cce\" (UID: \"2064a78b-5470-4026-991e-de1013ea2cce\") " Nov 01 20:45:25.255100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.254581 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2064a78b-5470-4026-991e-de1013ea2cce-kube-api-access-459l5\" (UniqueName: \"kubernetes.io/projected/2064a78b-5470-4026-991e-de1013ea2cce-kube-api-access-459l5\") pod \"2064a78b-5470-4026-991e-de1013ea2cce\" (UID: \"2064a78b-5470-4026-991e-de1013ea2cce\") " Nov 01 20:45:25.255350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.255149 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2064a78b-5470-4026-991e-de1013ea2cce-maas-api-service-ca" pod "2064a78b-5470-4026-991e-de1013ea2cce" (UID: "2064a78b-5470-4026-991e-de1013ea2cce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:25.280103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.277348 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2064a78b-5470-4026-991e-de1013ea2cce-kube-api-access-459l5" pod "2064a78b-5470-4026-991e-de1013ea2cce" (UID: "2064a78b-5470-4026-991e-de1013ea2cce"). InnerVolumeSpecName "kube-api-access-459l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:25.356692 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.356580 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2064a78b-5470-4026-991e-de1013ea2cce-maas-api-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:25.356692 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.356626 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459l5\" (UniqueName: \"kubernetes.io/projected/2064a78b-5470-4026-991e-de1013ea2cce-kube-api-access-459l5\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:25.753360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.753251 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" event={"ID":"2064a78b-5470-4026-991e-de1013ea2cce","Type":"ContainerDied","Data":"10dbb5094aff1c613e9a38346526c960ab104aae5bf5d0cd194d2d69cb0e0d6b"} Nov 01 20:45:25.753360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.753306 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d" Nov 01 20:45:25.753360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.753327 4477 scope.go:122] "RemoveContainer" containerID="59befdc56c8233214e21a33397aac04d46b3b0dce034e29cd81c1ba01b0c604f" Nov 01 20:45:25.873118 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.872513 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv"] Nov 01 20:45:25.927438 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.927337 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d"] Nov 01 20:45:25.931490 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:25.931455 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892765-wtq7d"] Nov 01 20:45:26.037500 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.037470 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:26.169973 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.169921 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fa9973-d0bc-48cd-a137-05a036138f13-kube-api-access-tjplr\" (UniqueName: \"kubernetes.io/projected/a9fa9973-d0bc-48cd-a137-05a036138f13-kube-api-access-tjplr\") pod \"a9fa9973-d0bc-48cd-a137-05a036138f13\" (UID: \"a9fa9973-d0bc-48cd-a137-05a036138f13\") " Nov 01 20:45:26.170197 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.170060 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fa9973-d0bc-48cd-a137-05a036138f13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa9973-d0bc-48cd-a137-05a036138f13-maas-api-service-ca\") pod \"a9fa9973-d0bc-48cd-a137-05a036138f13\" (UID: \"a9fa9973-d0bc-48cd-a137-05a036138f13\") " Nov 01 20:45:26.170885 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.170851 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fa9973-d0bc-48cd-a137-05a036138f13-maas-api-service-ca" pod "a9fa9973-d0bc-48cd-a137-05a036138f13" (UID: "a9fa9973-d0bc-48cd-a137-05a036138f13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:26.181527 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.181461 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fa9973-d0bc-48cd-a137-05a036138f13-kube-api-access-tjplr" pod "a9fa9973-d0bc-48cd-a137-05a036138f13" (UID: "a9fa9973-d0bc-48cd-a137-05a036138f13"). InnerVolumeSpecName "kube-api-access-tjplr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:26.272258 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.272163 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjplr\" (UniqueName: \"kubernetes.io/projected/a9fa9973-d0bc-48cd-a137-05a036138f13-kube-api-access-tjplr\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:26.272258 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.272204 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa9973-d0bc-48cd-a137-05a036138f13-maas-api-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:26.503786 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.503755 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2064a78b-5470-4026-991e-de1013ea2cce" path="/var/lib/kubelet/pods/2064a78b-5470-4026-991e-de1013ea2cce/volumes" Nov 01 20:45:26.760640 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.760612 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" Nov 01 20:45:26.760824 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.760612 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv" event={"ID":"a9fa9973-d0bc-48cd-a137-05a036138f13","Type":"ContainerDied","Data":"f817a8bae7351078ee1efaf094e5788e84d902563828970d2fa126236fa25ce5"} Nov 01 20:45:26.760824 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.760744 4477 scope.go:122] "RemoveContainer" containerID="3ad05d2853c6df9fb942cda659abf6ec007f198400c2a750d30e9d5db2167556" Nov 01 20:45:26.923222 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.923191 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv"] Nov 01 20:45:26.972412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:26.972377 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892765-9p4wv"] Nov 01 20:45:28.503001 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:28.502970 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" path="/var/lib/kubelet/pods/a9fa9973-d0bc-48cd-a137-05a036138f13/volumes" Nov 01 20:45:36.553310 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:36.553267 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:45:36.989771 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:36.989733 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk"] Nov 01 20:45:36.989976 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:36.989955 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" containerID="cri-o://e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95" gracePeriod=30 Nov 01 20:45:42.250791 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.250759 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:45:42.331142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331068 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-token\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-token\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331179 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-socket\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331211 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-kube-api-access-59b2g\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331242 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-credential-socket\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331314 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-podinfo\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331391 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57227b86-8401-4e61-8f01-daf8cdfbd753-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57227b86-8401-4e61-8f01-daf8cdfbd753-istiod-ca-cert\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331435 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-data\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331506 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-certs\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331537 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-socket" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:42.331669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331578 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-envoy\") pod \"57227b86-8401-4e61-8f01-daf8cdfbd753\" (UID: \"57227b86-8401-4e61-8f01-daf8cdfbd753\") " Nov 01 20:45:42.331953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331933 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.331953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331571 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-credential-socket" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:42.331953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331862 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-certs" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:42.332178 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.331905 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-data" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:42.332399 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.332279 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57227b86-8401-4e61-8f01-daf8cdfbd753-istiod-ca-cert" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:42.334026 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.333986 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-kube-api-access-59b2g" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "kube-api-access-59b2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:42.334211 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.334129 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-podinfo" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:45:42.334325 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.334303 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-token" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:42.334859 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.334823 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-envoy" pod "57227b86-8401-4e61-8f01-daf8cdfbd753" (UID: "57227b86-8401-4e61-8f01-daf8cdfbd753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:42.432963 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.432925 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-kube-api-access-59b2g\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.432963 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.432964 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.433194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.432979 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.433194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.432992 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57227b86-8401-4e61-8f01-daf8cdfbd753-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.433194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.433004 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.433194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.433017 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.433194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.433030 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.433194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.433041 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57227b86-8401-4e61-8f01-daf8cdfbd753-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:42.864224 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.864194 4477 generic.go:356] "Generic (PLEG): container finished" podID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerID="e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95" exitCode=0 Nov 01 20:45:42.864394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.864239 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" Nov 01 20:45:42.864394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.864279 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" event={"ID":"57227b86-8401-4e61-8f01-daf8cdfbd753","Type":"ContainerDied","Data":"e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95"} Nov 01 20:45:42.864394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.864342 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk" event={"ID":"57227b86-8401-4e61-8f01-daf8cdfbd753","Type":"ContainerDied","Data":"bd6d38acc3ba54223465de316700328f4669cd9abbc6ffb618edc5d73211d681"} Nov 01 20:45:42.864394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.864371 4477 scope.go:122] "RemoveContainer" containerID="e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95" Nov 01 20:45:42.873622 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.873604 4477 scope.go:122] "RemoveContainer" containerID="e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95" Nov 01 20:45:42.873919 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:45:42.873898 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95\": container with ID starting with e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95 not found: ID does not exist" containerID="e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95" Nov 01 20:45:42.874003 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:42.873932 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95"} err="failed to get container status \"e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95\": rpc error: code = NotFound desc = could not find container \"e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95\": container with ID starting with e4172a33bafd262328db1e8b3ff8408fcb70390ce822946a8b88249b26607e95 not found: ID does not exist" Nov 01 20:45:43.027399 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:43.027359 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk"] Nov 01 20:45:43.031749 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:43.031722 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-22bjk"] Nov 01 20:45:44.503145 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:44.503112 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" path="/var/lib/kubelet/pods/57227b86-8401-4e61-8f01-daf8cdfbd753/volumes" Nov 01 20:45:47.843156 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843119 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99"] Nov 01 20:45:47.843731 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843711 4477 cpu_manager.go:414] "containerMap: removing container" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843734 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843748 4477 cpu_manager.go:414] "containerMap: removing container" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843755 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843769 4477 cpu_manager.go:414] "containerMap: removing container" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843776 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843790 4477 cpu_manager.go:414] "containerMap: removing container" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843797 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843809 4477 cpu_manager.go:414] "containerMap: removing container" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:45:47.843832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843815 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843854 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843862 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843873 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.843879 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.844005 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.844021 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.844029 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.844040 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.844051 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.844059 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="57227b86-8401-4e61-8f01-daf8cdfbd753" containerName="istio-proxy" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.844069 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:45:47.844346 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.844097 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:45:47.849006 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.848986 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:47.893421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.893387 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:47.893540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.893428 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:47.893627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.893610 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hdjz\" (UniqueName: \"kubernetes.io/projected/6bda7899-dae6-4fef-b84c-da241fdbb894-kube-api-access-4hdjz\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:47.903555 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.903529 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99"] Nov 01 20:45:47.949589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.949562 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:45:47.949720 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.949601 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:45:47.994547 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.994513 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:47.994650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.994550 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:47.994650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.994626 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hdjz\" (UniqueName: \"kubernetes.io/projected/6bda7899-dae6-4fef-b84c-da241fdbb894-kube-api-access-4hdjz\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:47.995382 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.995351 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:47.997482 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:47.997442 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:48.051303 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:48.051265 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hdjz\" (UniqueName: \"kubernetes.io/projected/6bda7899-dae6-4fef-b84c-da241fdbb894-kube-api-access-4hdjz\") pod \"maas-api-revoke-sub-8210c8da-n6j99\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:48.162001 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:48.161917 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:48.539715 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:48.539689 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99"] Nov 01 20:45:48.541268 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:45:48.541241 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bda7899_dae6_4fef_b84c_da241fdbb894.slice/crio-ab8941db92c5f522160b3a9ac067d6a5ad0b8d6385978a60ccdf1b153c961b9a WatchSource:0}: Error finding container ab8941db92c5f522160b3a9ac067d6a5ad0b8d6385978a60ccdf1b153c961b9a: Status 404 returned error can't find the container with id ab8941db92c5f522160b3a9ac067d6a5ad0b8d6385978a60ccdf1b153c961b9a Nov 01 20:45:48.900497 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:48.900462 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" event={"ID":"6bda7899-dae6-4fef-b84c-da241fdbb894","Type":"ContainerStarted","Data":"9503bb938d3c072389d1e44ecd6d485b853686016801aa08000b14bcb1b82410"} Nov 01 20:45:48.900845 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:48.900522 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" event={"ID":"6bda7899-dae6-4fef-b84c-da241fdbb894","Type":"ContainerStarted","Data":"ab8941db92c5f522160b3a9ac067d6a5ad0b8d6385978a60ccdf1b153c961b9a"} Nov 01 20:45:49.013444 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:49.013301 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" podStartSLOduration=2.013283593 podStartE2EDuration="2.013283593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:49.012396661 +0000 UTC m=+902.662448206" watchObservedRunningTime="2026-11-01 20:45:49.013283593 +0000 UTC m=+902.663335140" Nov 01 20:45:49.908430 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:49.907948 4477 generic.go:356] "Generic (PLEG): container finished" podID="6bda7899-dae6-4fef-b84c-da241fdbb894" containerID="9503bb938d3c072389d1e44ecd6d485b853686016801aa08000b14bcb1b82410" exitCode=0 Nov 01 20:45:49.908430 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:49.907995 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" event={"ID":"6bda7899-dae6-4fef-b84c-da241fdbb894","Type":"ContainerDied","Data":"9503bb938d3c072389d1e44ecd6d485b853686016801aa08000b14bcb1b82410"} Nov 01 20:45:51.069002 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.068974 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:51.131108 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.131064 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bda7899-dae6-4fef-b84c-da241fdbb894-kube-api-access-4hdjz\" (UniqueName: \"kubernetes.io/projected/6bda7899-dae6-4fef-b84c-da241fdbb894-kube-api-access-4hdjz\") pod \"6bda7899-dae6-4fef-b84c-da241fdbb894\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " Nov 01 20:45:51.131108 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.131116 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-openshift-ca\") pod \"6bda7899-dae6-4fef-b84c-da241fdbb894\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " Nov 01 20:45:51.131392 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.131189 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-xks-ca\") pod \"6bda7899-dae6-4fef-b84c-da241fdbb894\" (UID: \"6bda7899-dae6-4fef-b84c-da241fdbb894\") " Nov 01 20:45:51.131544 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.131511 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-openshift-ca" pod "6bda7899-dae6-4fef-b84c-da241fdbb894" (UID: "6bda7899-dae6-4fef-b84c-da241fdbb894"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:51.133418 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.133392 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bda7899-dae6-4fef-b84c-da241fdbb894-kube-api-access-4hdjz" pod "6bda7899-dae6-4fef-b84c-da241fdbb894" (UID: "6bda7899-dae6-4fef-b84c-da241fdbb894"). InnerVolumeSpecName "kube-api-access-4hdjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:51.133603 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.133577 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-xks-ca" pod "6bda7899-dae6-4fef-b84c-da241fdbb894" (UID: "6bda7899-dae6-4fef-b84c-da241fdbb894"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:51.233180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.233065 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hdjz\" (UniqueName: \"kubernetes.io/projected/6bda7899-dae6-4fef-b84c-da241fdbb894-kube-api-access-4hdjz\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:51.233180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.233120 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-openshift-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:51.233180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.233131 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bda7899-dae6-4fef-b84c-da241fdbb894-maas-api-xks-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:51.920160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.920120 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" event={"ID":"6bda7899-dae6-4fef-b84c-da241fdbb894","Type":"ContainerDied","Data":"ab8941db92c5f522160b3a9ac067d6a5ad0b8d6385978a60ccdf1b153c961b9a"} Nov 01 20:45:51.920334 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.920163 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99" Nov 01 20:45:51.920334 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:45:51.920167 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8941db92c5f522160b3a9ac067d6a5ad0b8d6385978a60ccdf1b153c961b9a" Nov 01 20:46:01.954358 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954320 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97"] Nov 01 20:46:01.954784 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954769 4477 cpu_manager.go:414] "containerMap: removing container" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:46:01.954888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954785 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:46:01.954888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954800 4477 cpu_manager.go:414] "containerMap: removing container" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:46:01.954888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954806 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:46:01.954888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954834 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:46:01.954888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954840 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fa9973-d0bc-48cd-a137-05a036138f13" containerName="cleanup" Nov 01 20:46:01.954888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954857 4477 cpu_manager.go:414] "containerMap: removing container" podUID="6bda7899-dae6-4fef-b84c-da241fdbb894" containerName="revoke-keys" Nov 01 20:46:01.954888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954862 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bda7899-dae6-4fef-b84c-da241fdbb894" containerName="revoke-keys" Nov 01 20:46:01.955116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954929 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2064a78b-5470-4026-991e-de1013ea2cce" containerName="cleanup" Nov 01 20:46:01.955116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954942 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa2aeb7-e831-4fc6-a76e-7ce52ad53106" containerName="cleanup" Nov 01 20:46:01.955116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.954949 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bda7899-dae6-4fef-b84c-da241fdbb894" containerName="revoke-keys" Nov 01 20:46:01.958128 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:01.958112 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.010773 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.010747 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"openshift-service-ca.crt\"" Nov 01 20:46:02.011148 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.010836 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"kube-root-ca.crt\"" Nov 01 20:46:02.053574 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.053547 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-model-cache\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.053690 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.053590 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-dshm\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.053690 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.053644 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-home\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.053849 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.053834 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-tmp-dir\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.053909 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.053873 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5914cef2-d12a-4875-b1d2-02ead983c528-tls-certs\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.053991 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.053973 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzg6\" (UniqueName: \"kubernetes.io/projected/5914cef2-d12a-4875-b1d2-02ead983c528-kube-api-access-knzg6\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.060412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.060386 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"default-dockercfg-9d4fj\"" Nov 01 20:46:02.060518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.060424 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"distinct-w3-944263-kserve-self-signed-certs\"" Nov 01 20:46:02.062960 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.062941 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97"] Nov 01 20:46:02.154882 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.154848 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-tmp-dir\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.154882 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.154885 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5914cef2-d12a-4875-b1d2-02ead983c528-tls-certs\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.155158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.154928 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzg6\" (UniqueName: \"kubernetes.io/projected/5914cef2-d12a-4875-b1d2-02ead983c528-kube-api-access-knzg6\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.155158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.154974 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-model-cache\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.155158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.154998 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-dshm\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.155158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.155034 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-home\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.155462 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.155437 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-model-cache\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.155522 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.155462 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-home\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.155663 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.155644 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-tmp-dir\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.157569 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.157547 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-dshm\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.158171 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.158154 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5914cef2-d12a-4875-b1d2-02ead983c528-tls-certs\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.217022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.216943 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzg6\" (UniqueName: \"kubernetes.io/projected/5914cef2-d12a-4875-b1d2-02ead983c528-kube-api-access-knzg6\") pod \"distinct-w3-944263-kserve-55d477bd74-97p97\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.270731 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.270691 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:02.668799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.668768 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97"] Nov 01 20:46:02.669758 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:46:02.669715 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5914cef2_d12a_4875_b1d2_02ead983c528.slice/crio-f456d617deb6e85210c8832cb08ff5b1db2e7ffbeed399946b9630ea10d3188f WatchSource:0}: Error finding container f456d617deb6e85210c8832cb08ff5b1db2e7ffbeed399946b9630ea10d3188f: Status 404 returned error can't find the container with id f456d617deb6e85210c8832cb08ff5b1db2e7ffbeed399946b9630ea10d3188f Nov 01 20:46:02.987299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.987201 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" event={"ID":"5914cef2-d12a-4875-b1d2-02ead983c528","Type":"ContainerStarted","Data":"8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d"} Nov 01 20:46:02.987299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:02.987247 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" event={"ID":"5914cef2-d12a-4875-b1d2-02ead983c528","Type":"ContainerStarted","Data":"f456d617deb6e85210c8832cb08ff5b1db2e7ffbeed399946b9630ea10d3188f"} Nov 01 20:46:03.102378 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.102327 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podStartSLOduration=2.102313062 podStartE2EDuration="2.102313062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:03.099674135 +0000 UTC m=+916.749725681" watchObservedRunningTime="2026-11-01 20:46:03.102313062 +0000 UTC m=+916.752364607" Nov 01 20:46:03.270865 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.270777 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:03.765622 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.765589 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf"] Nov 01 20:46:03.769326 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.769307 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:03.821385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.821336 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:46:03.870557 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.870515 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:03.870717 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.870668 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fd8\" (UniqueName: \"kubernetes.io/projected/e2086947-04ed-4881-b707-20bb2121d3c5-kube-api-access-j7fd8\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:03.870717 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.870695 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:03.870810 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.870785 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:46:03.874223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.874199 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf"] Nov 01 20:46:03.971546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.971515 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:03.971723 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.971585 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fd8\" (UniqueName: \"kubernetes.io/projected/e2086947-04ed-4881-b707-20bb2121d3c5-kube-api-access-j7fd8\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:03.971723 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.971602 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:03.972371 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.972342 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:03.974056 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:03.974035 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:04.028520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:04.028456 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fd8\" (UniqueName: \"kubernetes.io/projected/e2086947-04ed-4881-b707-20bb2121d3c5-kube-api-access-j7fd8\") pod \"maas-api-revoke-sub-8eb25310-szhhf\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:04.083310 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:04.083284 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:04.460904 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:04.460877 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf"] Nov 01 20:46:04.462372 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:46:04.462345 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2086947_04ed_4881_b707_20bb2121d3c5.slice/crio-631a8bd44929c2322ef60c650711064d8d01b1158383e103dbf250a237bca4a8 WatchSource:0}: Error finding container 631a8bd44929c2322ef60c650711064d8d01b1158383e103dbf250a237bca4a8: Status 404 returned error can't find the container with id 631a8bd44929c2322ef60c650711064d8d01b1158383e103dbf250a237bca4a8 Nov 01 20:46:05.001164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:05.001022 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" event={"ID":"e2086947-04ed-4881-b707-20bb2121d3c5","Type":"ContainerStarted","Data":"5121710d35a1fe83266efaff25e0f09a7e459a47fecd8422a1a0b485898e11f6"} Nov 01 20:46:05.001164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:05.001110 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" event={"ID":"e2086947-04ed-4881-b707-20bb2121d3c5","Type":"ContainerStarted","Data":"631a8bd44929c2322ef60c650711064d8d01b1158383e103dbf250a237bca4a8"} Nov 01 20:46:05.114043 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:05.113975 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" podStartSLOduration=2.1139574899999998 podStartE2EDuration="2.11395749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:05.113351067 +0000 UTC m=+918.763402614" watchObservedRunningTime="2026-11-01 20:46:05.11395749 +0000 UTC m=+918.764009033" Nov 01 20:46:06.009179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:06.009141 4477 generic.go:356] "Generic (PLEG): container finished" podID="e2086947-04ed-4881-b707-20bb2121d3c5" containerID="5121710d35a1fe83266efaff25e0f09a7e459a47fecd8422a1a0b485898e11f6" exitCode=0 Nov 01 20:46:06.009350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:06.009231 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" event={"ID":"e2086947-04ed-4881-b707-20bb2121d3c5","Type":"ContainerDied","Data":"5121710d35a1fe83266efaff25e0f09a7e459a47fecd8422a1a0b485898e11f6"} Nov 01 20:46:06.335622 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:06.335587 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:46:06.553726 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:46:06.553690 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:46:07.199678 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.199650 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:07.319523 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.319492 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2086947-04ed-4881-b707-20bb2121d3c5-kube-api-access-j7fd8\" (UniqueName: \"kubernetes.io/projected/e2086947-04ed-4881-b707-20bb2121d3c5-kube-api-access-j7fd8\") pod \"e2086947-04ed-4881-b707-20bb2121d3c5\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " Nov 01 20:46:07.319752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.319684 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-xks-ca\") pod \"e2086947-04ed-4881-b707-20bb2121d3c5\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " Nov 01 20:46:07.319752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.319733 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-openshift-ca\") pod \"e2086947-04ed-4881-b707-20bb2121d3c5\" (UID: \"e2086947-04ed-4881-b707-20bb2121d3c5\") " Nov 01 20:46:07.320411 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.320364 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-openshift-ca" pod "e2086947-04ed-4881-b707-20bb2121d3c5" (UID: "e2086947-04ed-4881-b707-20bb2121d3c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:07.322727 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.322697 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-xks-ca" pod "e2086947-04ed-4881-b707-20bb2121d3c5" (UID: "e2086947-04ed-4881-b707-20bb2121d3c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:07.329516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.329483 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2086947-04ed-4881-b707-20bb2121d3c5-kube-api-access-j7fd8" pod "e2086947-04ed-4881-b707-20bb2121d3c5" (UID: "e2086947-04ed-4881-b707-20bb2121d3c5"). InnerVolumeSpecName "kube-api-access-j7fd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:07.421237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.421199 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-xks-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:07.421237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.421235 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2086947-04ed-4881-b707-20bb2121d3c5-maas-api-openshift-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:07.421237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:07.421249 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fd8\" (UniqueName: \"kubernetes.io/projected/e2086947-04ed-4881-b707-20bb2121d3c5-kube-api-access-j7fd8\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:08.051100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:08.050236 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" event={"ID":"e2086947-04ed-4881-b707-20bb2121d3c5","Type":"ContainerDied","Data":"631a8bd44929c2322ef60c650711064d8d01b1158383e103dbf250a237bca4a8"} Nov 01 20:46:08.051100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:08.050305 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="631a8bd44929c2322ef60c650711064d8d01b1158383e103dbf250a237bca4a8" Nov 01 20:46:08.051100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:08.050380 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf" Nov 01 20:46:12.271858 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:12.271815 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:12.285234 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:12.285208 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:13.093629 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:13.093597 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:46:36.553805 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:46:36.553715 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:46:37.337951 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.337919 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6"] Nov 01 20:46:37.338626 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.338601 4477 cpu_manager.go:414] "containerMap: removing container" podUID="e2086947-04ed-4881-b707-20bb2121d3c5" containerName="revoke-keys" Nov 01 20:46:37.338626 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.338622 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2086947-04ed-4881-b707-20bb2121d3c5" containerName="revoke-keys" Nov 01 20:46:37.338764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.338752 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2086947-04ed-4881-b707-20bb2121d3c5" containerName="revoke-keys" Nov 01 20:46:37.341264 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.341237 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.400084 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.400040 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6"] Nov 01 20:46:37.445444 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445404 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.445444 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445445 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq54w\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-kube-api-access-rq54w\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.445698 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445476 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.445698 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445605 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.445698 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445632 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.445698 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445655 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.445837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445778 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.445837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445810 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.445940 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.445889 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.546817 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.546780 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547023 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.546820 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547023 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.546848 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547023 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.546891 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547023 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.546922 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547023 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.546980 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.547337 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547476 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.547455 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547575 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.547488 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547575 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.547489 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq54w\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-kube-api-access-rq54w\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547575 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.547544 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547733 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.547718 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547974 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.547868 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.547974 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.547947 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.549867 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.549836 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.549968 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.549908 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.604471 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.604402 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq54w\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-kube-api-access-rq54w\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.604471 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.604403 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:37.657982 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:37.657950 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:38.138535 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:38.138472 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6"] Nov 01 20:46:38.140097 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:46:38.140050 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd279d92_f03e_4a2e_8a0d_b785efaeace6.slice/crio-294aaee8c45a078da79037b5312807c92950f33cdd4974e83b7b2d2c4557ec19 WatchSource:0}: Error finding container 294aaee8c45a078da79037b5312807c92950f33cdd4974e83b7b2d2c4557ec19: Status 404 returned error can't find the container with id 294aaee8c45a078da79037b5312807c92950f33cdd4974e83b7b2d2c4557ec19 Nov 01 20:46:38.142765 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:38.142727 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:38.142850 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:38.142799 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:38.142850 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:38.142830 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:38.254281 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:38.254245 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" event={"ID":"fd279d92-f03e-4a2e-8a0d-b785efaeace6","Type":"ContainerStarted","Data":"22dfeded88e18f8ed4238a4d1a184f985562ecc0974472b19ddbd5febadbcc83"} Nov 01 20:46:38.254457 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:38.254301 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" event={"ID":"fd279d92-f03e-4a2e-8a0d-b785efaeace6","Type":"ContainerStarted","Data":"294aaee8c45a078da79037b5312807c92950f33cdd4974e83b7b2d2c4557ec19"} Nov 01 20:46:38.373986 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:38.373929 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" podStartSLOduration=1.373914662 podStartE2EDuration="1.373914662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:38.371437081 +0000 UTC m=+952.021488627" watchObservedRunningTime="2026-11-01 20:46:38.373914662 +0000 UTC m=+952.023966543" Nov 01 20:46:38.658936 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:38.658891 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:39.659844 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:39.659805 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:46:39.660251 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:39.659871 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:46:39.972022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:39.971927 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:46:40.659107 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:40.659036 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:46:40.659282 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:40.659150 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:46:41.658892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:41.658849 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:46:41.659353 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:41.658935 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:46:42.658912 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:42.658871 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:46:42.659385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:42.658930 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:46:43.659373 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:43.659338 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:46:43.659857 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:43.659404 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:46:44.659105 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:44.659044 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:46:44.659291 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:44.659133 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:46:46.034596 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:46.034562 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:46.035046 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:46.034931 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:46.036002 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:46.035982 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 20:46:52.195117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.195084 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k"] Nov 01 20:46:52.197743 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.197727 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229371 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229344 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229402 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229479 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg6jj\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-kube-api-access-lg6jj\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229517 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229555 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229613 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b188ab-d523-42b9-b369-c3b2944d42d6-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229638 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229789 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229675 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.229789 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.229749 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.301950 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.301910 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k"] Nov 01 20:46:52.331015 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.330983 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331204 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331021 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg6jj\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-kube-api-access-lg6jj\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331204 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331064 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331204 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331127 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331204 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331173 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b188ab-d523-42b9-b369-c3b2944d42d6-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331204 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331198 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331554 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331251 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331554 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331285 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331554 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331319 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331715 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331553 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331715 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331633 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331820 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331751 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.331954 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331858 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.332039 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.331945 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b188ab-d523-42b9-b369-c3b2944d42d6-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.333692 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.333670 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.333779 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.333729 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.392062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.392027 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg6jj\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-kube-api-access-lg6jj\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.394894 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.394867 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.509529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.509454 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:52.976532 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:46:52.976503 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b188ab_d523_42b9_b369_c3b2944d42d6.slice/crio-5ea1074b0460e69dfa9f3851fe197a6de5b4e2e17a9da93275e75dded9251042 WatchSource:0}: Error finding container 5ea1074b0460e69dfa9f3851fe197a6de5b4e2e17a9da93275e75dded9251042: Status 404 returned error can't find the container with id 5ea1074b0460e69dfa9f3851fe197a6de5b4e2e17a9da93275e75dded9251042 Nov 01 20:46:52.976790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.976766 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k"] Nov 01 20:46:52.983134 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.983035 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:52.983271 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.983233 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:52.983348 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:52.983328 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:53.084124 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.084087 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8"] Nov 01 20:46:53.087439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.087412 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.141824 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.141791 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.141824 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.141830 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3207b3d-5929-483f-b6a8-704e28cb09c2-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.142129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.141858 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.142129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.141884 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.142129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.141996 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxpw7\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-kube-api-access-vxpw7\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.142357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.142148 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.142357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.142191 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.142357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.142225 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.142357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.142299 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.148176 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.148144 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8"] Nov 01 20:46:53.243687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.243586 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.243687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.243647 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.243687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.243684 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.244282 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.243765 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.244599 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244322 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.244742 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244625 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3207b3d-5929-483f-b6a8-704e28cb09c2-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.244742 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244654 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.244742 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244241 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.244742 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244569 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.244742 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244680 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.245111 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244787 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxpw7\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-kube-api-access-vxpw7\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.245111 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244250 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.245111 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.244170 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.245352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.245329 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3207b3d-5929-483f-b6a8-704e28cb09c2-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.247598 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.247574 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.248272 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.248240 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.300829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.300802 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.300988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.300967 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxpw7\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-kube-api-access-vxpw7\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.372115 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.372047 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" event={"ID":"e7b188ab-d523-42b9-b369-c3b2944d42d6","Type":"ContainerStarted","Data":"15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7"} Nov 01 20:46:53.372284 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.372144 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" event={"ID":"e7b188ab-d523-42b9-b369-c3b2944d42d6","Type":"ContainerStarted","Data":"5ea1074b0460e69dfa9f3851fe197a6de5b4e2e17a9da93275e75dded9251042"} Nov 01 20:46:53.409795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.409765 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:53.510014 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.509939 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:46:53.854600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.854548 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podStartSLOduration=1.854531521 podStartE2EDuration="1.854531521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:53.496163773 +0000 UTC m=+967.146215318" watchObservedRunningTime="2026-11-01 20:46:53.854531521 +0000 UTC m=+967.504583044" Nov 01 20:46:53.857466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.857441 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8"] Nov 01 20:46:53.858614 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:46:53.858577 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3207b3d_5929_483f_b6a8_704e28cb09c2.slice/crio-feec700378279d2097adc5aad718d97e27257146fd219c9793e59adfc9a6d72f WatchSource:0}: Error finding container feec700378279d2097adc5aad718d97e27257146fd219c9793e59adfc9a6d72f: Status 404 returned error can't find the container with id feec700378279d2097adc5aad718d97e27257146fd219c9793e59adfc9a6d72f Nov 01 20:46:53.861487 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.861455 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:53.861560 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.861521 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:53.861594 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:53.861567 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:46:54.381883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:54.381848 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" event={"ID":"b3207b3d-5929-483f-b6a8-704e28cb09c2","Type":"ContainerStarted","Data":"3efd509a119221d7c0f4d1a7d75da516102265711ed2e05109ce9acc94ab70a2"} Nov 01 20:46:54.382388 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:54.381903 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" event={"ID":"b3207b3d-5929-483f-b6a8-704e28cb09c2","Type":"ContainerStarted","Data":"feec700378279d2097adc5aad718d97e27257146fd219c9793e59adfc9a6d72f"} Nov 01 20:46:54.410871 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:54.410841 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:46:54.412197 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:54.412174 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:46:54.412317 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:54.412221 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:46:54.502861 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:54.502807 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podStartSLOduration=1.502790832 podStartE2EDuration="1.502790832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:54.500511326 +0000 UTC m=+968.150562872" watchObservedRunningTime="2026-11-01 20:46:54.502790832 +0000 UTC m=+968.152842375" Nov 01 20:46:54.512223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:54.512182 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:46:54.512352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:54.512241 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:46:55.410436 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:55.410404 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:46:55.410940 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:55.410915 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:46:55.510066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:55.510032 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:46:55.510245 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:55.510127 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:46:56.410580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:56.410544 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:46:56.411183 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:56.411154 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:46:56.514739 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:56.514708 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:46:56.514999 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:56.514974 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:46:57.410776 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:57.410729 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:46:57.411149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:57.410801 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:46:57.510844 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:57.510812 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:46:57.511012 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:57.510864 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:46:58.411092 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:58.411031 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:46:58.411482 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:58.411125 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:46:58.510757 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:58.510720 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:46:58.510900 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:58.510776 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:46:59.411153 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:59.411113 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:46:59.411567 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:59.411169 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:46:59.509968 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:59.509937 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:46:59.510154 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:46:59.509986 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:47:00.410627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:00.410587 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:47:00.410802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:00.410655 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:47:00.515228 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:00.515193 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:47:00.515883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:00.515623 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:47:00.516639 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:00.516618 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 20:47:01.415264 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:01.415233 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:47:01.415627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:01.415603 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:47:01.416304 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:01.416284 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 20:47:06.554164 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:47:06.554124 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:47:07.727967 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:07.727936 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj"] Nov 01 20:47:07.728511 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:07.728214 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" containerID="cri-o://9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51" gracePeriod=30 Nov 01 20:47:11.142166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:11.142124 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z2jrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:47:11.142749 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:11.142718 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:47:12.992583 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:12.992558 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:47:13.056892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.056798 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-credential-socket\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.056892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.056845 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-kube-api-access-tbbts\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-kube-api-access-tbbts\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.056892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.056878 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-socket\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.057237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.056914 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-token\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-token\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.057237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.056939 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-certs\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.057237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.056967 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d7cf7c-22e5-4201-852e-98b1b0944656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d7cf7c-22e5-4201-852e-98b1b0944656-istiod-ca-cert\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.057237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057015 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-data\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.057237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057121 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-envoy\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.057237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057184 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-credential-socket" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:13.057540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057280 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-socket" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:13.057540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057335 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-certs" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:13.057540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057363 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-podinfo\") pod \"80d7cf7c-22e5-4201-852e-98b1b0944656\" (UID: \"80d7cf7c-22e5-4201-852e-98b1b0944656\") " Nov 01 20:47:13.057540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057455 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-data" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:13.057540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057524 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d7cf7c-22e5-4201-852e-98b1b0944656-istiod-ca-cert" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:47:13.057915 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057896 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.057992 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057921 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.057992 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057936 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.057992 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057950 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d7cf7c-22e5-4201-852e-98b1b0944656-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.057992 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.057963 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.059683 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.059651 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-kube-api-access-tbbts" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "kube-api-access-tbbts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:47:13.059802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.059734 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-token" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:47:13.060064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.060044 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-podinfo" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:47:13.060146 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.060065 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-envoy" pod "80d7cf7c-22e5-4201-852e-98b1b0944656" (UID: "80d7cf7c-22e5-4201-852e-98b1b0944656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:13.159966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.159930 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.159966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.159958 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbts\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-kube-api-access-tbbts\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.159966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.159970 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.159966 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.159978 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d7cf7c-22e5-4201-852e-98b1b0944656-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:13.539551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.539519 4477 generic.go:356] "Generic (PLEG): container finished" podID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerID="9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51" exitCode=0 Nov 01 20:47:13.539752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.539568 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" Nov 01 20:47:13.539752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.539608 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" event={"ID":"80d7cf7c-22e5-4201-852e-98b1b0944656","Type":"ContainerDied","Data":"9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51"} Nov 01 20:47:13.539752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.539669 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj" event={"ID":"80d7cf7c-22e5-4201-852e-98b1b0944656","Type":"ContainerDied","Data":"a54f2ad6c705ed2d5d6c325eecc40835c7616771a0371ed5dc432af4004653fc"} Nov 01 20:47:13.539752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.539698 4477 scope.go:122] "RemoveContainer" containerID="9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51" Nov 01 20:47:13.550522 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.550505 4477 scope.go:122] "RemoveContainer" containerID="1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e" Nov 01 20:47:13.559981 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.559956 4477 scope.go:122] "RemoveContainer" containerID="9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51" Nov 01 20:47:13.560281 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:47:13.560264 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51\": container with ID starting with 9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51 not found: ID does not exist" containerID="9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51" Nov 01 20:47:13.560347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.560290 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51"} err="failed to get container status \"9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51\": rpc error: code = NotFound desc = could not find container \"9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51\": container with ID starting with 9b2046ea52726ce532fa57d70ef212b0623d9548d97dac6c3aa1dbd8f203cf51 not found: ID does not exist" Nov 01 20:47:13.560347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.560309 4477 scope.go:122] "RemoveContainer" containerID="1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e" Nov 01 20:47:13.560620 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:47:13.560596 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e\": container with ID starting with 1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e not found: ID does not exist" containerID="1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e" Nov 01 20:47:13.560676 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.560630 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e"} err="failed to get container status \"1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e\": rpc error: code = NotFound desc = could not find container \"1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e\": container with ID starting with 1d3b128e85437dab890455700ff57cba5e8490e7914f875d0750ac64df8ad88e not found: ID does not exist" Nov 01 20:47:13.704716 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.704685 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj"] Nov 01 20:47:13.754024 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:13.753989 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z2jrj"] Nov 01 20:47:14.504544 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:14.504508 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" path="/var/lib/kubelet/pods/80d7cf7c-22e5-4201-852e-98b1b0944656/volumes" Nov 01 20:47:22.951455 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:22.951421 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv"] Nov 01 20:47:22.952130 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:22.952111 4477 cpu_manager.go:414] "containerMap: removing container" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" Nov 01 20:47:22.952198 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:22.952133 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" Nov 01 20:47:22.952278 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:22.952267 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" Nov 01 20:47:22.952314 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:22.952286 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" Nov 01 20:47:22.956311 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:22.956291 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.008624 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.008597 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-dd038b-openshift-default-dockercfg-vkx72\"" Nov 01 20:47:23.062410 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.062377 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv"] Nov 01 20:47:23.067897 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.067867 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.068043 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.067928 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.068127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.068033 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.068226 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.068201 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.068271 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.068252 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adba17bf-847c-4f95-a0d6-34c71729277e-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.068403 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.068381 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.068445 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.068418 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9fq\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-kube-api-access-vw9fq\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.068491 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.068445 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.068614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.068596 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adba17bf-847c-4f95-a0d6-34c71729277e-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.169941 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.169901 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adba17bf-847c-4f95-a0d6-34c71729277e-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.169955 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.169978 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.169995 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170039 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170106 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adba17bf-847c-4f95-a0d6-34c71729277e-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170454 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170172 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170454 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170208 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9fq\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-kube-api-access-vw9fq\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170454 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170236 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170632 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170479 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170632 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170542 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170632 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170587 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.170797 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.170781 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.171449 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.171423 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adba17bf-847c-4f95-a0d6-34c71729277e-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.172732 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.172711 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.172839 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.172823 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adba17bf-847c-4f95-a0d6-34c71729277e-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.227029 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.226944 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.227249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.227112 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9fq\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-kube-api-access-vw9fq\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.269166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.269131 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:23.702275 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.702240 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv"] Nov 01 20:47:23.703896 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:47:23.703874 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadba17bf_847c_4f95_a0d6_34c71729277e.slice/crio-dca5b440efdb9fea8f21854c16d20891804b49d8098d2fed19281c0f70b13c78 WatchSource:0}: Error finding container dca5b440efdb9fea8f21854c16d20891804b49d8098d2fed19281c0f70b13c78: Status 404 returned error can't find the container with id dca5b440efdb9fea8f21854c16d20891804b49d8098d2fed19281c0f70b13c78 Nov 01 20:47:23.706362 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.706330 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:47:23.706426 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.706397 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:47:23.706461 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:23.706428 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:47:24.136943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:24.136904 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:47:24.611892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:24.610960 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" event={"ID":"adba17bf-847c-4f95-a0d6-34c71729277e","Type":"ContainerStarted","Data":"d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11"} Nov 01 20:47:24.612067 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:24.611935 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" event={"ID":"adba17bf-847c-4f95-a0d6-34c71729277e","Type":"ContainerStarted","Data":"dca5b440efdb9fea8f21854c16d20891804b49d8098d2fed19281c0f70b13c78"} Nov 01 20:47:24.786473 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:24.786402 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" podStartSLOduration=2.78636776 podStartE2EDuration="2.78636776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:24.729741085 +0000 UTC m=+998.379792631" watchObservedRunningTime="2026-11-01 20:47:24.78636776 +0000 UTC m=+998.436419306" Nov 01 20:47:25.278707 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:25.278671 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:25.280582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:25.280544 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:47:25.280734 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:25.280622 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:47:26.270384 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:26.270341 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:47:26.270619 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:26.270416 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:47:27.270472 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:27.270438 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:47:27.270911 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:27.270495 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:47:28.269902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:28.269849 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:47:28.270141 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:28.269924 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:47:29.269822 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:29.269785 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:47:29.270370 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:29.269851 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:47:30.269448 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:30.269415 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:47:30.269640 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:30.269471 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:47:31.273646 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:31.273612 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:31.274062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:31.274025 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:31.274763 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:31.274745 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 20:47:33.630425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.630368 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv"] Nov 01 20:47:33.631144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.631126 4477 cpu_manager.go:414] "containerMap: removing container" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" Nov 01 20:47:33.631199 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.631147 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d7cf7c-22e5-4201-852e-98b1b0944656" containerName="istio-proxy" Nov 01 20:47:33.634943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.634921 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.690553 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.690507 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv"] Nov 01 20:47:33.735438 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.735404 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"default-dockercfg-rfgc8\"" Nov 01 20:47:33.735587 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.735427 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"kube-root-ca.crt\"" Nov 01 20:47:33.736884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.736859 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"openshift-service-ca.crt\"" Nov 01 20:47:33.736884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.736865 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"distinct-w5-c9ae21-kserve-self-signed-certs\"" Nov 01 20:47:33.737131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.736905 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7v85\" (UniqueName: \"kubernetes.io/projected/ab94912a-4370-4155-965b-e4661b7a75d2-kube-api-access-j7v85\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.737131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.736937 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-home\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.737131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.737036 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-model-cache\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.737131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.737103 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-dshm\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.737358 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.737157 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-tmp-dir\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.737358 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.737224 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94912a-4370-4155-965b-e4661b7a75d2-tls-certs\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.838700 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.838661 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94912a-4370-4155-965b-e4661b7a75d2-tls-certs\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.838907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.838772 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7v85\" (UniqueName: \"kubernetes.io/projected/ab94912a-4370-4155-965b-e4661b7a75d2-kube-api-access-j7v85\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.838907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.838808 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-home\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.838907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.838869 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-model-cache\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.839123 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.838934 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-dshm\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.839123 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.838998 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-tmp-dir\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.839497 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.839472 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-tmp-dir\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.839701 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.839678 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-home\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.839838 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.839791 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-model-cache\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.842163 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.842132 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-dshm\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.842265 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.842218 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94912a-4370-4155-965b-e4661b7a75d2-tls-certs\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.897024 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.896961 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7v85\" (UniqueName: \"kubernetes.io/projected/ab94912a-4370-4155-965b-e4661b7a75d2-kube-api-access-j7v85\") pod \"distinct-w5-c9ae21-kserve-595b75d75f-wjtdv\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:33.950204 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:33.950166 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:34.384904 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:34.384879 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv"] Nov 01 20:47:34.386788 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:47:34.386759 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab94912a_4370_4155_965b_e4661b7a75d2.slice/crio-c31b1b61bb1e369e6aa9bc15aca5a758834c0c9a2e5b2df9d80c170f32d6a0a1 WatchSource:0}: Error finding container c31b1b61bb1e369e6aa9bc15aca5a758834c0c9a2e5b2df9d80c170f32d6a0a1: Status 404 returned error can't find the container with id c31b1b61bb1e369e6aa9bc15aca5a758834c0c9a2e5b2df9d80c170f32d6a0a1 Nov 01 20:47:34.705061 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:34.704971 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" event={"ID":"ab94912a-4370-4155-965b-e4661b7a75d2","Type":"ContainerStarted","Data":"35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48"} Nov 01 20:47:34.705061 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:34.705023 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" event={"ID":"ab94912a-4370-4155-965b-e4661b7a75d2","Type":"ContainerStarted","Data":"c31b1b61bb1e369e6aa9bc15aca5a758834c0c9a2e5b2df9d80c170f32d6a0a1"} Nov 01 20:47:34.823482 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:34.823416 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podStartSLOduration=1.823397327 podStartE2EDuration="1.823397327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:34.818989962 +0000 UTC m=+1008.469041501" watchObservedRunningTime="2026-11-01 20:47:34.823397327 +0000 UTC m=+1008.473448873" Nov 01 20:47:34.951035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:34.950528 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:36.553941 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:47:36.553899 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:47:43.950689 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:43.950656 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:43.963339 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:43.963306 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:44.795300 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:44.795274 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:47:53.564778 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.564740 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p"] Nov 01 20:47:53.570139 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.570114 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.674810 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.674767 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.674987 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.674901 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6726a-f88f-430f-a5ed-cd29c37edc98-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.675087 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.675052 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.675153 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.675135 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.675196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.675173 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbsf\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-kube-api-access-zwbsf\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.675314 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.675249 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.675314 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.675304 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.675422 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.675355 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.675422 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.675410 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776274 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6726a-f88f-430f-a5ed-cd29c37edc98-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776395 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776440 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776471 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbsf\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-kube-api-access-zwbsf\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776506 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776530 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776586 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776631 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.776719 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.777260 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.777945 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6726a-f88f-430f-a5ed-cd29c37edc98-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.778238 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.779180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.778505 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.780253 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.780220 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.781993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.781964 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.783679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.783649 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.786236 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.786205 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p"] Nov 01 20:47:53.835224 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.835153 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbsf\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-kube-api-access-zwbsf\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.836374 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.836344 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:53.883581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:53.883544 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:54.318852 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:54.318820 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p"] Nov 01 20:47:54.320693 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:47:54.320668 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b6726a_f88f_430f_a5ed_cd29c37edc98.slice/crio-42ac04e7ef12de2ede69d285759269471c6c30950ff08248f21c1fbd8c8454fc WatchSource:0}: Error finding container 42ac04e7ef12de2ede69d285759269471c6c30950ff08248f21c1fbd8c8454fc: Status 404 returned error can't find the container with id 42ac04e7ef12de2ede69d285759269471c6c30950ff08248f21c1fbd8c8454fc Nov 01 20:47:54.323064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:54.323026 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:47:54.323182 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:54.323147 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:47:54.323246 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:54.323198 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:47:54.859666 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:54.859627 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" event={"ID":"31b6726a-f88f-430f-a5ed-cd29c37edc98","Type":"ContainerStarted","Data":"baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652"} Nov 01 20:47:54.860160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:54.859692 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" event={"ID":"31b6726a-f88f-430f-a5ed-cd29c37edc98","Type":"ContainerStarted","Data":"42ac04e7ef12de2ede69d285759269471c6c30950ff08248f21c1fbd8c8454fc"} Nov 01 20:47:54.883986 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:54.883956 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:47:55.089207 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:55.089148 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podStartSLOduration=2.08913502 podStartE2EDuration="2.08913502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:55.037911717 +0000 UTC m=+1028.687963286" watchObservedRunningTime="2026-11-01 20:47:55.08913502 +0000 UTC m=+1028.739186565" Nov 01 20:47:55.884989 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:55.884958 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:47:55.885396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:55.885010 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:47:56.884239 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:56.884137 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:47:56.884239 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:56.884214 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:47:57.884442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:57.884405 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:47:57.884834 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:57.884465 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:47:58.884365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:58.884328 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:47:58.884550 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:58.884405 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:47:59.884890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:59.884851 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:47:59.885344 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:47:59.884912 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:48:00.884915 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:00.884866 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:48:00.885368 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:00.884941 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:48:01.889254 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:01.889226 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:48:01.889716 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:01.889682 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:48:01.890488 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:01.890470 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 20:48:03.050637 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:03.050607 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:48:06.554989 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:48:06.554952 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:48:06.866519 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:06.866491 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8"] Nov 01 20:48:06.870429 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:06.870408 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:06.926637 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:06.926598 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8"] Nov 01 20:48:06.971277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:06.971241 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"distinct-w2-cd248f-kserve-self-signed-certs\"" Nov 01 20:48:07.064101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.063350 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-home\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.064101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.063410 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dpr\" (UniqueName: \"kubernetes.io/projected/f09423e7-1763-473d-b671-729f2d634515-kube-api-access-k4dpr\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.064101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.063498 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-tmp-dir\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.064101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.063541 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-dshm\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.064101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.063598 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09423e7-1763-473d-b671-729f2d634515-tls-certs\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.064101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.063630 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-model-cache\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.164449 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.164370 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-home\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.164449 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.164412 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dpr\" (UniqueName: \"kubernetes.io/projected/f09423e7-1763-473d-b671-729f2d634515-kube-api-access-k4dpr\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.164684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.164487 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-tmp-dir\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.164684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.164525 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-dshm\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.164684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.164579 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09423e7-1763-473d-b671-729f2d634515-tls-certs\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.164684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.164610 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-model-cache\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.164948 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.164911 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-tmp-dir\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.165100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.164948 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-model-cache\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.165207 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.165181 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-home\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.167179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.167153 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-dshm\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.167466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.167445 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09423e7-1763-473d-b671-729f2d634515-tls-certs\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.220912 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.220881 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dpr\" (UniqueName: \"kubernetes.io/projected/f09423e7-1763-473d-b671-729f2d634515-kube-api-access-k4dpr\") pod \"distinct-w2-cd248f-kserve-68cbd7474d-jgbv8\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.485329 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.485241 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:07.867995 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.867967 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8"] Nov 01 20:48:07.869300 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:48:07.869273 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf09423e7_1763_473d_b671_729f2d634515.slice/crio-de24b3cc0d322c07642cd10a06e61b74fb86dffa8a9c4e469c198701a2ad37d5 WatchSource:0}: Error finding container de24b3cc0d322c07642cd10a06e61b74fb86dffa8a9c4e469c198701a2ad37d5: Status 404 returned error can't find the container with id de24b3cc0d322c07642cd10a06e61b74fb86dffa8a9c4e469c198701a2ad37d5 Nov 01 20:48:07.978290 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.978255 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" event={"ID":"f09423e7-1763-473d-b671-729f2d634515","Type":"ContainerStarted","Data":"3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b"} Nov 01 20:48:07.978480 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:07.978307 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" event={"ID":"f09423e7-1763-473d-b671-729f2d634515","Type":"ContainerStarted","Data":"de24b3cc0d322c07642cd10a06e61b74fb86dffa8a9c4e469c198701a2ad37d5"} Nov 01 20:48:08.094419 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:08.094371 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podStartSLOduration=2.094352276 podStartE2EDuration="2.094352276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:48:08.09090212 +0000 UTC m=+1041.740953669" watchObservedRunningTime="2026-11-01 20:48:08.094352276 +0000 UTC m=+1041.744403823" Nov 01 20:48:08.485843 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:08.485807 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:09.637781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.637735 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8"] Nov 01 20:48:09.642775 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.642752 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.692927 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.692891 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.693149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.692936 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.693149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.692959 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.693149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.693112 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.693267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.693159 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52zm\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-kube-api-access-d52zm\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.693267 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.693242 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30768f6-fbd9-4d98-adcd-59455e939e31-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.693335 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.693286 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.693335 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.693301 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.693335 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.693324 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.694977 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.694943 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1a1e9e-openshift-default-dockercfg-xvvfj\"" Nov 01 20:48:09.749855 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.749813 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8"] Nov 01 20:48:09.793957 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.793927 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.793957 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.793963 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794234 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.793992 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794234 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794029 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52zm\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-kube-api-access-d52zm\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794234 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794062 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30768f6-fbd9-4d98-adcd-59455e939e31-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794389 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794228 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794389 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794268 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794389 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794297 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794545 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794439 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794545 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794449 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794545 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794443 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794713 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.794788 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.794718 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.795193 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.795166 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30768f6-fbd9-4d98-adcd-59455e939e31-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.796671 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.796634 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.797174 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.797153 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.851372 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.851338 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.851543 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.851392 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52zm\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-kube-api-access-d52zm\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:09.958214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:09.958124 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:10.391121 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:10.391091 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8"] Nov 01 20:48:10.393264 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:48:10.393237 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30768f6_fbd9_4d98_adcd_59455e939e31.slice/crio-86c19e6ebdccb929fa2a40a1dab617dd84edb28770dcea00fe0e73a9df0e9093 WatchSource:0}: Error finding container 86c19e6ebdccb929fa2a40a1dab617dd84edb28770dcea00fe0e73a9df0e9093: Status 404 returned error can't find the container with id 86c19e6ebdccb929fa2a40a1dab617dd84edb28770dcea00fe0e73a9df0e9093 Nov 01 20:48:10.395679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:10.395646 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:48:10.395772 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:10.395709 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:48:10.395772 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:10.395740 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:48:11.012889 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:11.012847 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" event={"ID":"b30768f6-fbd9-4d98-adcd-59455e939e31","Type":"ContainerStarted","Data":"7f507be80f83ce139a543634f6a24ffc5846a970a10212ad0b79dde149023ef8"} Nov 01 20:48:11.014019 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:11.012955 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" event={"ID":"b30768f6-fbd9-4d98-adcd-59455e939e31","Type":"ContainerStarted","Data":"86c19e6ebdccb929fa2a40a1dab617dd84edb28770dcea00fe0e73a9df0e9093"} Nov 01 20:48:11.250101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:11.249599 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" podStartSLOduration=2.249577837 podStartE2EDuration="2.249577837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:48:11.178319381 +0000 UTC m=+1044.828370939" watchObservedRunningTime="2026-11-01 20:48:11.249577837 +0000 UTC m=+1044.899629383" Nov 01 20:48:11.958704 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:11.958664 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:11.960279 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:11.960253 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 20:48:11.960397 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:11.960303 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 20:48:12.959422 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:12.959385 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 20:48:12.959820 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:12.959444 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 20:48:13.958626 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:13.958590 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 20:48:13.958848 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:13.958644 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 20:48:14.959217 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:14.959180 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 20:48:14.959611 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:14.959234 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 20:48:15.958838 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:15.958798 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 20:48:15.959028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:15.958857 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 20:48:16.958925 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:16.958884 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 20:48:16.959371 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:16.958963 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 20:48:17.485880 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:17.485843 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:17.499572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:17.499542 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:17.962815 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:17.962787 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:17.963246 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:17.963170 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:17.963769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:17.963747 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 20:48:18.095898 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:18.095869 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:48:36.559913 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:48:36.559878 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:48:41.758766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:48:41.758727 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:49:06.597862 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:49:06.595357 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:49:12.021951 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:12.021913 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:49:23.388340 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.388306 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4"] Nov 01 20:49:23.392541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.392519 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.493515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493479 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.493705 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493553 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.493705 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493662 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.493832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493706 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.493832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493803 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.493832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493826 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.493998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493865 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.493998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493966 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.494158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.493998 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9b5m\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-kube-api-access-b9b5m\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.596996 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.596812 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.596996 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.596870 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.596996 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.596937 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.596996 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.596985 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.597375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.597015 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9b5m\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-kube-api-access-b9b5m\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.597375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.597140 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.597375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.597245 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.597375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.597298 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.597375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.597347 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.597646 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.597471 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4"] Nov 01 20:49:23.597842 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.597817 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.598201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.598178 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.598402 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.598375 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.598671 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.598647 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.599296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.599259 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.601578 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.601554 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.606863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.606835 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.654338 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.654248 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9b5m\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-kube-api-access-b9b5m\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.654566 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.654544 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:23.708381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:23.708342 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:24.150245 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.150215 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4"] Nov 01 20:49:24.152745 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:49:24.152717 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6667ff8_33fa_4bad_956a_1c9ae709b0bc.slice/crio-da072ba6dc402445e4f88370518e119979ed5e9f61b2c8cabaf57eebbf28d47b WatchSource:0}: Error finding container da072ba6dc402445e4f88370518e119979ed5e9f61b2c8cabaf57eebbf28d47b: Status 404 returned error can't find the container with id da072ba6dc402445e4f88370518e119979ed5e9f61b2c8cabaf57eebbf28d47b Nov 01 20:49:24.155700 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.155658 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:24.155807 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.155744 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:24.155807 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.155787 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:24.239909 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.239872 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48"] Nov 01 20:49:24.244420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.244396 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.296262 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.296219 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"distinct-2-w1-db0157-kserve-self-signed-certs\"" Nov 01 20:49:24.299761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.299729 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48"] Nov 01 20:49:24.306122 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.306067 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672pv\" (UniqueName: \"kubernetes.io/projected/3abd0466-122f-449d-a2e9-82d6f408ef11-kube-api-access-672pv\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.306274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.306183 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-dshm\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.306274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.306230 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-model-cache\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.306274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.306270 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-tmp-dir\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.306437 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.306384 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abd0466-122f-449d-a2e9-82d6f408ef11-tls-certs\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.306495 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.306453 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-home\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.408309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.408210 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-dshm\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.408309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.408280 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-model-cache\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.408804 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.408330 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-tmp-dir\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.408804 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.408403 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abd0466-122f-449d-a2e9-82d6f408ef11-tls-certs\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.408804 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.408502 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-home\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.408804 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.408582 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672pv\" (UniqueName: \"kubernetes.io/projected/3abd0466-122f-449d-a2e9-82d6f408ef11-kube-api-access-672pv\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.409641 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.409611 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-model-cache\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.409752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.409662 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-home\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.410022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.409994 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-tmp-dir\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.414543 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.412723 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abd0466-122f-449d-a2e9-82d6f408ef11-tls-certs\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.414993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.414970 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-dshm\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.468655 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.468622 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-672pv\" (UniqueName: \"kubernetes.io/projected/3abd0466-122f-449d-a2e9-82d6f408ef11-kube-api-access-672pv\") pod \"distinct-2-w1-db0157-kserve-67c894bb6-8rv48\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.555099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.550436 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9"] Nov 01 20:49:24.555476 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.555451 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.567629 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.567590 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:24.585485 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.585445 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" event={"ID":"e6667ff8-33fa-4bad-956a-1c9ae709b0bc","Type":"ContainerStarted","Data":"1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe"} Nov 01 20:49:24.585636 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.585521 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" event={"ID":"e6667ff8-33fa-4bad-956a-1c9ae709b0bc","Type":"ContainerStarted","Data":"da072ba6dc402445e4f88370518e119979ed5e9f61b2c8cabaf57eebbf28d47b"} Nov 01 20:49:24.612264 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.612139 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.612264 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.612217 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a5139-633b-492a-8c56-4edeedce6f97-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.631160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.612807 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.631160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.612865 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.631160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.612960 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.631160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.613033 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m264v\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-kube-api-access-m264v\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.631160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.613116 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a5139-633b-492a-8c56-4edeedce6f97-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.656603 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.656266 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.656603 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.656346 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.694189 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.687266 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9"] Nov 01 20:49:24.711224 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.711190 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:24.764615 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.764558 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.764884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.764630 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.764884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.764707 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.764884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.764741 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a5139-633b-492a-8c56-4edeedce6f97-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.764884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.764781 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.764884 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.764814 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.765396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.765274 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.765555 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.765537 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.765680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.765619 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.765773 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.765735 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m264v\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-kube-api-access-m264v\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.765859 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.765820 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a5139-633b-492a-8c56-4edeedce6f97-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.765960 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.765861 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.767106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.767066 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.769255 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.769233 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a5139-633b-492a-8c56-4edeedce6f97-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.773590 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.773567 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.781679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.781638 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a5139-633b-492a-8c56-4edeedce6f97-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.814159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.814113 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podStartSLOduration=1.8140978639999998 podStartE2EDuration="1.814097864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:24.811757389 +0000 UTC m=+1118.461808953" watchObservedRunningTime="2026-11-01 20:49:24.814097864 +0000 UTC m=+1118.464149410" Nov 01 20:49:24.850289 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.849530 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.852026 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.851826 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m264v\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-kube-api-access-m264v\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:24.874547 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:24.874511 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:25.100020 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.099993 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48"] Nov 01 20:49:25.100474 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:49:25.100452 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3abd0466_122f_449d_a2e9_82d6f408ef11.slice/crio-c4b52518766d96cf98538deeb211411ec233c73dbf7b20630be45445ee5b735c WatchSource:0}: Error finding container c4b52518766d96cf98538deeb211411ec233c73dbf7b20630be45445ee5b735c: Status 404 returned error can't find the container with id c4b52518766d96cf98538deeb211411ec233c73dbf7b20630be45445ee5b735c Nov 01 20:49:25.265891 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.265859 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9"] Nov 01 20:49:25.271572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.271533 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:25.271684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.271606 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:25.271814 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.271798 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:25.594572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.594539 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" event={"ID":"3abd0466-122f-449d-a2e9-82d6f408ef11","Type":"ContainerStarted","Data":"697ef9e510b42fe7c82800cc7899e487ea8b6cb28f5ce307044a9c8aa6202175"} Nov 01 20:49:25.594572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.594578 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" event={"ID":"3abd0466-122f-449d-a2e9-82d6f408ef11","Type":"ContainerStarted","Data":"c4b52518766d96cf98538deeb211411ec233c73dbf7b20630be45445ee5b735c"} Nov 01 20:49:25.596661 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.596621 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" event={"ID":"921a5139-633b-492a-8c56-4edeedce6f97","Type":"ContainerStarted","Data":"3122c958df5be523e7f158a94de52086f2e3c274bbbebef2d3f00c501a385dbd"} Nov 01 20:49:25.596785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.596672 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" event={"ID":"921a5139-633b-492a-8c56-4edeedce6f97","Type":"ContainerStarted","Data":"bd13bcdaa67bd940e219c47cf88507e120f2eed8c93294844d80035f2c03f9eb"} Nov 01 20:49:25.710344 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.710263 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:49:25.710344 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.710311 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 20:49:25.754460 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.753147 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podStartSLOduration=1.7531271149999998 podStartE2EDuration="1.753127115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:25.737396212 +0000 UTC m=+1119.387447760" watchObservedRunningTime="2026-11-01 20:49:25.753127115 +0000 UTC m=+1119.403178658" Nov 01 20:49:25.859309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.859208 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podStartSLOduration=1.859187646 podStartE2EDuration="1.859187646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:25.858635925 +0000 UTC m=+1119.508687470" watchObservedRunningTime="2026-11-01 20:49:25.859187646 +0000 UTC m=+1119.509239192" Nov 01 20:49:25.875411 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:25.875374 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:26.577274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:26.575708 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:26.709270 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:26.709232 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:49:26.709666 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:26.709288 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 20:49:26.878139 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:26.878036 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 20:49:26.878384 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:26.878355 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 20:49:27.709349 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:27.709306 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:49:27.709841 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:27.709357 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 20:49:27.877761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:27.875470 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 20:49:27.877761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:27.875530 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 20:49:28.709657 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:28.709624 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:49:28.710184 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:28.709669 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 20:49:28.879096 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:28.878611 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 20:49:28.879096 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:28.878698 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 20:49:29.708763 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:29.708723 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:49:29.708946 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:29.708775 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 20:49:29.876101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:29.875629 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 20:49:29.876101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:29.875729 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 20:49:30.725129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:30.711574 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:49:30.725129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:30.711645 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 20:49:30.881457 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:30.881264 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 20:49:30.881457 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:30.881349 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 20:49:31.709525 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:31.709481 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:49:31.709715 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:31.709555 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 20:49:31.875567 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:31.875531 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 20:49:31.875741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:31.875612 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 20:49:32.715398 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:32.715360 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:32.716281 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:32.716058 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:32.716882 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:32.716861 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 20:49:33.230840 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:33.230805 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:33.231587 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:33.231512 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:33.233021 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:33.233002 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:49:34.568486 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:34.568452 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:34.582193 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:34.582163 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:34.726259 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:34.726229 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:49:36.554198 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:49:36.554157 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:49:46.522826 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:46.522796 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:49:49.085386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:49.085320 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.091849 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:49.091147 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.132100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:49.130207 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:54.115284 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.115250 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr"] Nov 01 20:49:54.119335 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.119316 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284344 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284579 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284407 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284579 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284488 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284699 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284679 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284715 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284796 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284785 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284835 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284811 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284900 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284885 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkfk\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-kube-api-access-kfkfk\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.284952 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.284942 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c772c60-c4a9-4ed9-9215-f7996d644ede-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.327673 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.327639 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr"] Nov 01 20:49:54.386475 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386389 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.386475 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386426 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.386689 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386623 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.386747 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386686 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.386747 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386732 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkfk\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-kube-api-access-kfkfk\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.386868 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386808 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c772c60-c4a9-4ed9-9215-f7996d644ede-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.386868 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386848 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.386976 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386849 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.386976 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386938 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.387112 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.386967 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.387197 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.387180 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.387301 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.387283 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.387837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.387321 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.388179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.388146 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c772c60-c4a9-4ed9-9215-f7996d644ede-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.392795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.392770 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.396837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.396808 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.443445 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.443404 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.443620 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.443520 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkfk\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-kube-api-access-kfkfk\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:54.733531 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:54.733438 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:55.175956 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.175925 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr"] Nov 01 20:49:55.177257 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:49:55.177235 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c772c60_c4a9_4ed9_9215_f7996d644ede.slice/crio-cd3b7442d5ab338eacab18fb67fd4bfc3e896cf264242a71857336a2a762a7f4 WatchSource:0}: Error finding container cd3b7442d5ab338eacab18fb67fd4bfc3e896cf264242a71857336a2a762a7f4: Status 404 returned error can't find the container with id cd3b7442d5ab338eacab18fb67fd4bfc3e896cf264242a71857336a2a762a7f4 Nov 01 20:49:55.179745 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.179710 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:55.179845 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.179791 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:55.179845 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.179836 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:49:55.392540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.392506 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm"] Nov 01 20:49:55.397421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.397382 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.453792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.453687 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"distinct-2-w5-c9ae21-kserve-self-signed-certs\"" Nov 01 20:49:55.459604 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.459555 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm"] Nov 01 20:49:55.502652 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.502613 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-home\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.503049 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.503012 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-tmp-dir\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.503185 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.503105 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2fefb38-b099-4c25-a5ec-1934adac9046-tls-certs\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.503185 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.503160 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbbp\" (UniqueName: \"kubernetes.io/projected/b2fefb38-b099-4c25-a5ec-1934adac9046-kube-api-access-mkbbp\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.503295 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.503213 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-dshm\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.503367 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.503342 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-model-cache\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.605550 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.605511 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-home\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.605764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.605590 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-tmp-dir\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.605764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.605617 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2fefb38-b099-4c25-a5ec-1934adac9046-tls-certs\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.605764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.605647 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbbp\" (UniqueName: \"kubernetes.io/projected/b2fefb38-b099-4c25-a5ec-1934adac9046-kube-api-access-mkbbp\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.605764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.605698 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-dshm\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.605764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.605757 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-model-cache\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.606092 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.606040 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-tmp-dir\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.606181 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.606068 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-home\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.606237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.606216 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-model-cache\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.608880 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.608844 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-dshm\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.609102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.608898 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2fefb38-b099-4c25-a5ec-1934adac9046-tls-certs\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.667121 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.663395 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbbp\" (UniqueName: \"kubernetes.io/projected/b2fefb38-b099-4c25-a5ec-1934adac9046-kube-api-access-mkbbp\") pod \"distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.720376 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.720290 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:55.948132 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.947607 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" event={"ID":"9c772c60-c4a9-4ed9-9215-f7996d644ede","Type":"ContainerStarted","Data":"edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d"} Nov 01 20:49:55.948132 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:55.947718 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" event={"ID":"9c772c60-c4a9-4ed9-9215-f7996d644ede","Type":"ContainerStarted","Data":"cd3b7442d5ab338eacab18fb67fd4bfc3e896cf264242a71857336a2a762a7f4"} Nov 01 20:49:56.068228 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:56.068172 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podStartSLOduration=2.068158195 podStartE2EDuration="2.068158195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:56.06480304 +0000 UTC m=+1149.714854585" watchObservedRunningTime="2026-11-01 20:49:56.068158195 +0000 UTC m=+1149.718209810" Nov 01 20:49:56.228243 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:56.228204 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm"] Nov 01 20:49:56.238703 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:49:56.238668 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2fefb38_b099_4c25_a5ec_1934adac9046.slice/crio-4a5eb2d67b761222e0022a93de2a3dc66a06e43a08b230fc334fd341ff3fe3cb WatchSource:0}: Error finding container 4a5eb2d67b761222e0022a93de2a3dc66a06e43a08b230fc334fd341ff3fe3cb: Status 404 returned error can't find the container with id 4a5eb2d67b761222e0022a93de2a3dc66a06e43a08b230fc334fd341ff3fe3cb Nov 01 20:49:56.751736 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:56.751287 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:49:56.753065 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:56.752671 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:49:56.753065 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:56.752725 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:49:56.958428 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:56.958385 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" event={"ID":"b2fefb38-b099-4c25-a5ec-1934adac9046","Type":"ContainerStarted","Data":"97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253"} Nov 01 20:49:56.958670 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:56.958649 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" event={"ID":"b2fefb38-b099-4c25-a5ec-1934adac9046","Type":"ContainerStarted","Data":"4a5eb2d67b761222e0022a93de2a3dc66a06e43a08b230fc334fd341ff3fe3cb"} Nov 01 20:49:57.730102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:57.725172 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:49:57.740099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:57.735481 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:49:57.740099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:57.735547 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:49:58.741518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:58.741472 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:49:58.742031 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:58.741535 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:49:59.736512 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:59.736469 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:49:59.736818 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:49:59.736784 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:50:00.739129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:00.738619 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:50:00.739129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:00.738684 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:50:01.734656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:01.734620 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:50:01.734847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:01.734687 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:50:02.849136 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:02.849056 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:50:02.849671 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:02.849635 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:50:02.850681 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:02.850662 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 20:50:03.017342 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:03.017279 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podStartSLOduration=8.017261785 podStartE2EDuration="8.017261785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:57.072408412 +0000 UTC m=+1150.722459957" watchObservedRunningTime="2026-11-01 20:50:03.017261785 +0000 UTC m=+1156.667313364" Nov 01 20:50:05.721211 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:05.721175 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:50:05.734402 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:05.734375 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:50:06.038608 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:06.038504 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:50:06.560102 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:50:06.556642 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:50:09.056504 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:09.056470 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt"] Nov 01 20:50:09.057031 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:09.056726 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" containerID="cri-o://2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24" gracePeriod=30 Nov 01 20:50:11.583142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.583091 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8rkmx-tkvvw"] Nov 01 20:50:11.587408 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.587378 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.642363 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.642321 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8rkmx-tkvvw"] Nov 01 20:50:11.687783 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.687745 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 20:50:11.736124 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.736057 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-trusted-ca-bundle\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.736304 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.736153 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-archives-path\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.736304 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.736259 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dqfd\" (UniqueName: \"kubernetes.io/projected/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-kube-api-access-6dqfd\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.736387 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.736313 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-service-ca-bundle\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.737601 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.737580 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9m2k9\"" Nov 01 20:50:11.789469 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.789441 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 20:50:11.837736 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.837638 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-service-ca-bundle\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.837736 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.837725 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-trusted-ca-bundle\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.837978 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.837780 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-archives-path\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.837978 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.837926 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dqfd\" (UniqueName: \"kubernetes.io/projected/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-kube-api-access-6dqfd\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.838250 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.838221 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-archives-path\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.838478 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.838456 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-service-ca-bundle\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.838873 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.838851 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-trusted-ca-bundle\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.894845 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.894810 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dqfd\" (UniqueName: \"kubernetes.io/projected/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-kube-api-access-6dqfd\") pod \"periodic-gathering-8rkmx-tkvvw\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:11.901533 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:11.901503 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:50:12.302266 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:12.302242 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8rkmx-tkvvw"] Nov 01 20:50:12.303915 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:50:12.303888 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190501a4_91d5_43b1_96ef_b9f9ca6bde4b.slice/crio-a16e0edb63f678e05a142df50f52233256851860ac3c74575b004d19ea385e78 WatchSource:0}: Error finding container a16e0edb63f678e05a142df50f52233256851860ac3c74575b004d19ea385e78: Status 404 returned error can't find the container with id a16e0edb63f678e05a142df50f52233256851860ac3c74575b004d19ea385e78 Nov 01 20:50:13.099807 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:13.099571 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" event={"ID":"190501a4-91d5-43b1-96ef-b9f9ca6bde4b","Type":"ContainerStarted","Data":"d84fb203b715f607a17f03c2183c3c52eeba08e0e1088126686338cf916ef225"} Nov 01 20:50:13.111277 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:13.100525 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" event={"ID":"190501a4-91d5-43b1-96ef-b9f9ca6bde4b","Type":"ContainerStarted","Data":"a16e0edb63f678e05a142df50f52233256851860ac3c74575b004d19ea385e78"} Nov 01 20:50:13.233789 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:13.233717 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" podStartSLOduration=2.233697051 podStartE2EDuration="2.233697051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:13.227603462 +0000 UTC m=+1166.877655007" watchObservedRunningTime="2026-11-01 20:50:13.233697051 +0000 UTC m=+1166.883748597" Nov 01 20:50:13.786101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:13.769338 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pcwvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:50:13.786101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:13.769405 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:50:14.601003 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.600973 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:50:14.628576 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.628543 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-token\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-token\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.628780 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.628760 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istiod-ca-cert\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.628858 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.628813 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-podinfo\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.628930 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.628917 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-certs\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.629027 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629014 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-envoy\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.629116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629102 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-kube-api-access-kqg26\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-kube-api-access-kqg26\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.629191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629176 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-socket\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.629248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629183 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istiod-ca-cert" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:14.629248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629236 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-credential-socket\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.629379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629296 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-data\") pod \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\" (UID: \"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e\") " Nov 01 20:50:14.629379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629315 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-certs" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:14.629512 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629458 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-socket" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:14.629571 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629507 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-credential-socket" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:14.629708 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.629686 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-data" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:14.630054 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.630033 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:14.630158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.630063 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:14.630158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.630110 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:14.630158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.630126 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:14.630158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.630140 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:14.631632 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.631597 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-kube-api-access-kqg26" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "kube-api-access-kqg26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:14.631906 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.631875 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-podinfo" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:50:14.632013 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.631915 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-token" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:14.632707 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.632680 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-envoy" pod "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" (UID: "a67e64c1-5c69-473e-8e56-d9c6be9c6d2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:14.731721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.731630 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqg26\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-kube-api-access-kqg26\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:14.731721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.731661 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:14.731721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.731674 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:14.731721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:14.731682 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:15.200660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.200618 4477 generic.go:356] "Generic (PLEG): container finished" podID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerID="2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24" exitCode=0 Nov 01 20:50:15.200849 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.200737 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" Nov 01 20:50:15.200927 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.200850 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" event={"ID":"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e","Type":"ContainerDied","Data":"2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24"} Nov 01 20:50:15.200927 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.200897 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt" event={"ID":"a67e64c1-5c69-473e-8e56-d9c6be9c6d2e","Type":"ContainerDied","Data":"df8aa6e8754fc107f322c4fb38a625a0d8d5f6848fecf117e7c712ff1f0fcb25"} Nov 01 20:50:15.201043 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.200926 4477 scope.go:122] "RemoveContainer" containerID="2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24" Nov 01 20:50:15.214296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.214277 4477 scope.go:122] "RemoveContainer" containerID="2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24" Nov 01 20:50:15.214572 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:50:15.214553 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24\": container with ID starting with 2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24 not found: ID does not exist" containerID="2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24" Nov 01 20:50:15.214668 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.214580 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24"} err="failed to get container status \"2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24\": rpc error: code = NotFound desc = could not find container \"2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24\": container with ID starting with 2aea13bca5ff112680181bd0cd76949da2d0e5e30dfe9104c66d69a375908d24 not found: ID does not exist" Nov 01 20:50:15.365385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.365350 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt"] Nov 01 20:50:15.415296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:15.415255 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pcwvt"] Nov 01 20:50:16.504722 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:16.504694 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" path="/var/lib/kubelet/pods/a67e64c1-5c69-473e-8e56-d9c6be9c6d2e/volumes" Nov 01 20:50:17.785230 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.785191 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf"] Nov 01 20:50:17.785709 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.785696 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" Nov 01 20:50:17.785766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.785710 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" Nov 01 20:50:17.785832 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.785821 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67e64c1-5c69-473e-8e56-d9c6be9c6d2e" containerName="istio-proxy" Nov 01 20:50:17.790489 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.790471 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.846660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.846628 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf"] Nov 01 20:50:17.870115 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.870060 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gs4j\" (UniqueName: \"kubernetes.io/projected/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-kube-api-access-6gs4j\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.870288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.870127 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tmp-dir\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.870328 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.870306 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tls-certs\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.870389 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.870371 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-dshm\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.870453 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.870435 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-home\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.870518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.870464 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-model-cache\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.891304 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.891276 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"distinct-2-w2-cd248f-kserve-self-signed-certs\"" Nov 01 20:50:17.972396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.972356 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gs4j\" (UniqueName: \"kubernetes.io/projected/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-kube-api-access-6gs4j\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.972618 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.972415 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tmp-dir\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.972618 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.972596 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tls-certs\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.972748 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.972674 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-dshm\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.972748 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.972718 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-home\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.972959 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.972932 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tmp-dir\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.973152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.973028 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-home\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.973152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.973106 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-model-cache\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.973501 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.973474 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-model-cache\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.975794 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.975770 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-dshm\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:17.976171 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:17.976150 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tls-certs\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:18.029249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:18.029219 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gs4j\" (UniqueName: \"kubernetes.io/projected/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-kube-api-access-6gs4j\") pod \"distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:18.104548 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:18.104274 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:18.545516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:18.545488 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf"] Nov 01 20:50:18.546263 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:50:18.546243 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4c3028b_28a8_4a1c_9ce6_99ac8ab5622e.slice/crio-af1bf8e20fa97298b8388efb815e42704b0efb5f9c6e6e13d9b08134cbac0d5a WatchSource:0}: Error finding container af1bf8e20fa97298b8388efb815e42704b0efb5f9c6e6e13d9b08134cbac0d5a: Status 404 returned error can't find the container with id af1bf8e20fa97298b8388efb815e42704b0efb5f9c6e6e13d9b08134cbac0d5a Nov 01 20:50:19.298871 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:19.298834 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" event={"ID":"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e","Type":"ContainerStarted","Data":"2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512"} Nov 01 20:50:19.298871 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:19.298884 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" event={"ID":"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e","Type":"ContainerStarted","Data":"af1bf8e20fa97298b8388efb815e42704b0efb5f9c6e6e13d9b08134cbac0d5a"} Nov 01 20:50:19.413179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:19.413127 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podStartSLOduration=2.413111759 podStartE2EDuration="2.413111759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:19.410555866 +0000 UTC m=+1173.060607407" watchObservedRunningTime="2026-11-01 20:50:19.413111759 +0000 UTC m=+1173.063163298" Nov 01 20:50:20.105248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:20.105206 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:25.694175 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.694139 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv"] Nov 01 20:50:25.699319 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.699300 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.772879 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.772840 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.773051 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.772984 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.773051 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.773025 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.773149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.773066 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.773189 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.773152 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.773223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.773209 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.773317 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.773300 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gshs9\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-kube-api-access-gshs9\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.773411 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.773395 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.773449 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.773428 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874336 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874301 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874336 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874341 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874370 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874408 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gshs9\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-kube-api-access-gshs9\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874483 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874508 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874614 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874559 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874632 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874654 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.874998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.874942 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.875063 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.875012 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.875148 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.875095 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.875243 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.875223 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.875299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.875276 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.876997 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.876975 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.877117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.877062 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.902595 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.902557 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv"] Nov 01 20:50:25.931634 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.931598 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:25.931821 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:25.931710 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gshs9\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-kube-api-access-gshs9\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:26.017735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:26.017649 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:26.456499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:26.456467 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv"] Nov 01 20:50:26.456827 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:50:26.456804 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba27e49_0af7_4f7e_8e2e_1ecf2190cc13.slice/crio-5c5f84c4d2747209bb83b7ec0d262a5602e7ffeb8b3f80ecdeba29c3b32cd1d4 WatchSource:0}: Error finding container 5c5f84c4d2747209bb83b7ec0d262a5602e7ffeb8b3f80ecdeba29c3b32cd1d4: Status 404 returned error can't find the container with id 5c5f84c4d2747209bb83b7ec0d262a5602e7ffeb8b3f80ecdeba29c3b32cd1d4 Nov 01 20:50:26.459825 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:26.459794 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:50:26.459899 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:26.459857 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:50:26.459947 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:26.459900 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:50:27.362995 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:27.360669 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" event={"ID":"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13","Type":"ContainerStarted","Data":"eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2"} Nov 01 20:50:27.362995 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:27.360763 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" event={"ID":"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13","Type":"ContainerStarted","Data":"5c5f84c4d2747209bb83b7ec0d262a5602e7ffeb8b3f80ecdeba29c3b32cd1d4"} Nov 01 20:50:27.549104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:27.548336 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podStartSLOduration=2.548313851 podStartE2EDuration="2.548313851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:27.526594745 +0000 UTC m=+1181.176646315" watchObservedRunningTime="2026-11-01 20:50:27.548313851 +0000 UTC m=+1181.198365397" Nov 01 20:50:28.028097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:28.021772 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:28.028097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:28.021949 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 20:50:28.028097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:28.022013 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 20:50:28.104911 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:28.104876 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:28.150221 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:28.150191 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:28.379755 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:28.379728 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:29.034101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:29.029174 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 20:50:29.034101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:29.029253 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 20:50:30.022101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:30.020512 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 20:50:30.022101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:30.020591 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 20:50:31.036104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:31.032715 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 20:50:31.036104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:31.032786 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 20:50:32.018243 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:32.018197 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 20:50:32.018459 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:32.018264 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 20:50:33.018294 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:33.018255 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 20:50:33.018713 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:33.018324 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 20:50:34.023244 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:34.023213 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:34.023732 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:34.023712 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:34.024421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:34.024399 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:50:34.613380 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:34.613345 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:50:36.722114 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:36.720186 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:36.728402 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:36.728359 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:36.795260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:36.795201 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:36.819400 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:50:36.819344 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:50:36.884393 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:36.884338 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.020292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.020183 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.086223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.086178 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.090182 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.090146 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.090793 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.090765 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 20:50:37.380103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.379165 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.752776 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.752665 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.872106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.769169 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.872106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.865610 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:50:37.872106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.862171 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.872106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.866956 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:50:37.894114 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.889190 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.894114 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:37.889672 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:50:38.052176 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.035865 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 01 20:50:38.052176 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.040733 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:50:38.098142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.082236 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:50:38.099116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.098499 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:38.099116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.098936 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:50:38.110366 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.108707 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:50:38.110366 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.108782 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:50:38.159124 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.155196 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:38.159124 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.155657 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:50:38.363653 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.363622 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:50:38.364701 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.364682 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:50:38.385159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.385123 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:38.385703 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.385685 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:38.491910 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.491879 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:50:38.883101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.882145 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:50:38.883101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.882230 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:50:38.883101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.882268 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:50:39.000098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:38.971100 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:50:52.045783 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:52.045748 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99"] Nov 01 20:50:52.095324 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:52.095286 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8210c8da-n6j99"] Nov 01 20:50:52.504943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:52.504913 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bda7899-dae6-4fef-b84c-da241fdbb894" path="/var/lib/kubelet/pods/6bda7899-dae6-4fef-b84c-da241fdbb894/volumes" Nov 01 20:50:55.521081 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.521051 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth"] Nov 01 20:50:55.527023 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.527005 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.579048 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.579018 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9ae21-openshift-default-dockercfg-k8hz9\"" Nov 01 20:50:55.607835 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.607806 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.607986 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.607875 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.608033 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.607996 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.608109 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.608038 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbtk\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-kube-api-access-6dbtk\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.608177 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.608161 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.608233 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.608217 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.608359 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.608342 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.608398 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.608380 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.608604 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.608589 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.637258 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.637221 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth"] Nov 01 20:50:55.710010 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.709975 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710031 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710115 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710149 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbtk\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-kube-api-access-6dbtk\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710168 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710196 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710431 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710249 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710431 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710280 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710431 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710312 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.710856 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710832 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.711029 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710920 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.711029 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.710936 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.711235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.711059 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.711235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.711184 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.712853 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.712830 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.713149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.713133 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.766291 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.766257 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.766546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.766518 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbtk\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-kube-api-access-6dbtk\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:55.840878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:55.840848 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:56.277018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:56.276986 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth"] Nov 01 20:50:56.278151 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:50:56.278121 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48e99fc_dcb4_41b8_9ae7_95422d0f9a23.slice/crio-e249afbb24b4e98f9ca14d0159bc79813752a28d6f87fd0591ffd80c94e8eb23 WatchSource:0}: Error finding container e249afbb24b4e98f9ca14d0159bc79813752a28d6f87fd0591ffd80c94e8eb23: Status 404 returned error can't find the container with id e249afbb24b4e98f9ca14d0159bc79813752a28d6f87fd0591ffd80c94e8eb23 Nov 01 20:50:56.280599 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:56.280562 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:50:56.280715 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:56.280652 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:50:56.280715 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:56.280700 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:50:56.977904 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:56.977805 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" event={"ID":"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23","Type":"ContainerStarted","Data":"c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873"} Nov 01 20:50:56.977904 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:56.977869 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" event={"ID":"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23","Type":"ContainerStarted","Data":"e249afbb24b4e98f9ca14d0159bc79813752a28d6f87fd0591ffd80c94e8eb23"} Nov 01 20:50:57.098920 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:57.098839 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podStartSLOduration=2.098818791 podStartE2EDuration="2.098818791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:57.098496475 +0000 UTC m=+1210.748548021" watchObservedRunningTime="2026-11-01 20:50:57.098818791 +0000 UTC m=+1210.748870336" Nov 01 20:50:57.842447 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:57.842416 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:50:57.845173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:57.845134 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:50:57.845347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:57.845190 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 20:50:58.843463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:58.843425 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:50:58.843987 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:58.843492 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 20:50:59.841491 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:59.841449 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:50:59.842500 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:50:59.842465 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 20:51:00.841631 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:00.841589 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:51:00.842144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:00.841654 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 20:51:01.841677 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:01.841637 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:51:01.842315 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:01.842257 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 20:51:02.841812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:02.841763 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:51:02.842400 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:02.841836 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 20:51:03.847288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:03.847255 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:51:03.847808 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:03.847590 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:51:03.848516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:03.848480 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:51:06.556342 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:51:06.556302 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:51:08.047405 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:08.047364 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf"] Nov 01 20:51:08.097623 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:08.097584 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb25310-szhhf"] Nov 01 20:51:08.506201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:08.506164 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2086947-04ed-4881-b707-20bb2121d3c5" path="/var/lib/kubelet/pods/e2086947-04ed-4881-b707-20bb2121d3c5/volumes" Nov 01 20:51:09.421793 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.421758 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth"] Nov 01 20:51:09.425937 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.425921 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507190 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507149 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507282 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507325 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4chs5\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-kube-api-access-4chs5\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507385 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507425 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507500 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507530 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326305c7-129f-4b07-9c0d-6fb1aff5167b-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507745 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507608 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.507796 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.507761 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609020 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.608986 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609020 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609019 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4chs5\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-kube-api-access-4chs5\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609046 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609069 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609121 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609149 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326305c7-129f-4b07-9c0d-6fb1aff5167b-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609633 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609484 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609633 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609535 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609633 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609554 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609633 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609594 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609833 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609708 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609833 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609761 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.609953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609923 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326305c7-129f-4b07-9c0d-6fb1aff5167b-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.610022 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.609977 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.611702 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.611679 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.612198 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.612180 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.624666 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.624643 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth"] Nov 01 20:51:09.674221 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.674149 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4chs5\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-kube-api-access-4chs5\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.674561 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.674541 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.746113 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.743329 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:09.791225 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.760969 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6"] Nov 01 20:51:09.804240 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.803123 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.858852 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.853940 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6"] Nov 01 20:51:09.933131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933050 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.933307 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933215 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.933307 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933274 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.933307 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933297 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.933478 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933341 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.933478 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933410 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.933478 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933431 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.933630 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933484 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97tm\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-kube-api-access-d97tm\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:09.933630 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:09.933574 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.034698 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.034659 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.034922 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.034734 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.034922 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.034765 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.034922 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.034812 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.034922 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.034882 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.034922 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.034906 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.035229 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.034969 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97tm\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-kube-api-access-d97tm\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.035229 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.035056 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.035229 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.035146 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.035651 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.035619 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.035941 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.035918 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.036268 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.036234 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.036393 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.036357 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.037106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.037054 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.038548 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.038513 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.040036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.040010 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.092948 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.092904 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.094535 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.094501 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97tm\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-kube-api-access-d97tm\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.175379 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.175338 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:10.321997 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.321959 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth"] Nov 01 20:51:10.357927 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:51:10.357878 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326305c7_129f_4b07_9c0d_6fb1aff5167b.slice/crio-db7198a3172cbc301197b97404de2a67df9d36120dfc02d68a2f3b1003ff4340 WatchSource:0}: Error finding container db7198a3172cbc301197b97404de2a67df9d36120dfc02d68a2f3b1003ff4340: Status 404 returned error can't find the container with id db7198a3172cbc301197b97404de2a67df9d36120dfc02d68a2f3b1003ff4340 Nov 01 20:51:10.362547 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.361581 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:10.362547 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.361676 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:10.362547 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.361751 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:10.364827 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.364696 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k"] Nov 01 20:51:10.377112 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.374189 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.380681 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.380653 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt"] Nov 01 20:51:10.391059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.390965 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.445866 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.444757 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k"] Nov 01 20:51:10.453290 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.453203 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt"] Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661232 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661308 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661415 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661537 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661607 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661638 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661690 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661730 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c85304-e51f-434e-a787-bb4315eabe39-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661761 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbw8\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-kube-api-access-mqbw8\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661791 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8tv\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-kube-api-access-lk8tv\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661816 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661861 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661891 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.661946 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.662006 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c85304-e51f-434e-a787-bb4315eabe39-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.662093 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cc78a3-f034-4873-9e85-0481a94d6818-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.662132 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.662452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.662163 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785321 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785374 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785460 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785501 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785559 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785612 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785650 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785683 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785735 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785778 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c85304-e51f-434e-a787-bb4315eabe39-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785810 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbw8\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-kube-api-access-mqbw8\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785840 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8tv\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-kube-api-access-lk8tv\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785866 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785903 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785932 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.785988 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.786052 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c85304-e51f-434e-a787-bb4315eabe39-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.786139 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cc78a3-f034-4873-9e85-0481a94d6818-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.817013 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cc78a3-f034-4873-9e85-0481a94d6818-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.818387 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.818677 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.818914 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.819165 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.865167 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.865247 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.865293 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.866449 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.877562 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.878116 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.879350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.878379 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.918500 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.903144 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c85304-e51f-434e-a787-bb4315eabe39-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:10.918500 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:10.906050 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c85304-e51f-434e-a787-bb4315eabe39-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:11.129762 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.129549 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6"] Nov 01 20:51:11.149984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.149447 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:11.149984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.149862 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:11.149984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.149935 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:11.174312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.168481 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:11.174312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.169147 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:11.174312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.171935 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8tv\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-kube-api-access-lk8tv\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:11.174312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.172190 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:11.219299 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.211472 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbw8\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-kube-api-access-mqbw8\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:11.518347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.508458 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:11.594174 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.594135 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh"] Nov 01 20:51:11.662529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.662471 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:11.741159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.737701 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh"] Nov 01 20:51:11.741159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.737913 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.746781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.743618 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:11.791547 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:51:11.791496 4477 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd69fd9_9a03_42be_b4d5_6ab1513cfd5c.slice/crio-0688182a850706f0766858beba3d7658277e3e0161a3eecfa8029339121feaaf.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.854695 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.854826 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.854871 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.855022 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.855058 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.855155 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.855208 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.855282 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlr5\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-kube-api-access-ttlr5\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.855351 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.871442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.866438 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:11.904133 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.901143 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" event={"ID":"326305c7-129f-4b07-9c0d-6fb1aff5167b","Type":"ContainerStarted","Data":"db7198a3172cbc301197b97404de2a67df9d36120dfc02d68a2f3b1003ff4340"} Nov 01 20:51:11.911628 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.911593 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" event={"ID":"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c","Type":"ContainerStarted","Data":"54cdf7bd902cca82ab1095899f34b1ffa5c74d9654324cc96a0fd9fdce08f91e"} Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956350 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956415 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956477 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlr5\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-kube-api-access-ttlr5\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956537 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956578 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956685 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956725 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956856 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.956898 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.957784 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.958095 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.961090 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.960224 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.961090 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.960502 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:11.961090 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:11.960749 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:12.078193 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.069498 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:12.078193 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.069999 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:12.117795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.093047 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podStartSLOduration=3.09302743 podStartE2EDuration="3.09302743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:12.07960284 +0000 UTC m=+1225.729654385" watchObservedRunningTime="2026-11-01 20:51:12.09302743 +0000 UTC m=+1225.743078975" Nov 01 20:51:12.204152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.203708 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:12.204152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.203831 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:12.204152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.203873 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:12.237826 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.237761 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podStartSLOduration=3.23773761 podStartE2EDuration="3.23773761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:12.21046117 +0000 UTC m=+1225.860512752" watchObservedRunningTime="2026-11-01 20:51:12.23773761 +0000 UTC m=+1225.887789157" Nov 01 20:51:12.966160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.908195 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 20:51:12.966160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.931986 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlr5\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-kube-api-access-ttlr5\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:12.966160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:12.938346 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:13.061439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.061390 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:13.259055 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.258966 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:13.413104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.374536 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:13.555343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.375027 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:51:13.555343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.378188 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:51:13.555343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.378896 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:13.555343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.522672 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:13.555343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.433788 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:13.555343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.522763 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:13.628975 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:51:13.628929 4477 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.13s" Nov 01 20:51:13.629402 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.629385 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:13.629663 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.629648 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt"] Nov 01 20:51:13.629874 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.629861 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:51:13.630038 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.630027 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:51:13.630252 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.630240 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:51:13.630354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.630342 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k"] Nov 01 20:51:13.808497 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.808403 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:13.808988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.808959 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:13.809752 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.809725 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:13.810296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.808455 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:13.810452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.810432 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:13.813201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.813158 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:13.813418 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.813379 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:13.813615 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.813575 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:13.830767 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.830720 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" event={"ID":"326305c7-129f-4b07-9c0d-6fb1aff5167b","Type":"ContainerStarted","Data":"ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae"} Nov 01 20:51:13.846193 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.846155 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" event={"ID":"f3cc78a3-f034-4873-9e85-0481a94d6818","Type":"ContainerStarted","Data":"437b22b35bf8a202e96e5625a219a599018c060c52abc916265bffd4aa293bea"} Nov 01 20:51:13.848729 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.848698 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" event={"ID":"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c","Type":"ContainerStarted","Data":"0688182a850706f0766858beba3d7658277e3e0161a3eecfa8029339121feaaf"} Nov 01 20:51:13.908782 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.908444 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:13.945144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:13.941460 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:14.115270 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.115228 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:51:14.195288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.195167 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:14.195288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.195225 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:14.748159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.747755 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:14.748159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.747814 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:14.845111 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.844965 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:14.845766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.845707 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:14.852134 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.846014 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:14.903239 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:14.895593 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh"] Nov 01 20:51:15.027829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.027792 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" event={"ID":"fd61b1b0-c480-4678-8e78-4e68d4b8b120","Type":"ContainerStarted","Data":"094ceda89bf0e84bf57cbe20794a31fc3ae309f648152720b5338988c11d1c5b"} Nov 01 20:51:15.031238 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.031206 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" event={"ID":"f3cc78a3-f034-4873-9e85-0481a94d6818","Type":"ContainerStarted","Data":"ce01ca70074e110490301b0840eaeb5ee10c170fcb4780ffbd37f620316b5519"} Nov 01 20:51:15.195193 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.191909 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:15.195193 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.194678 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:15.195193 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.194803 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:15.195758 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.195287 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:15.195758 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.195337 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:15.202620 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.202570 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" event={"ID":"49c85304-e51f-434e-a787-bb4315eabe39","Type":"ContainerStarted","Data":"2661f25d453e0c232205a887303de3008a15aec20511a0fb45996db6bb02fcbb"} Nov 01 20:51:15.202785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.202644 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" event={"ID":"49c85304-e51f-434e-a787-bb4315eabe39","Type":"ContainerStarted","Data":"5faabdb7d9dec6b8af9f5bc6d276e98176c3ac6344202c42489adf6da16644b0"} Nov 01 20:51:15.297512 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.297450 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podStartSLOduration=5.297427673 podStartE2EDuration="5.297427673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:15.279907519 +0000 UTC m=+1228.929959312" watchObservedRunningTime="2026-11-01 20:51:15.297427673 +0000 UTC m=+1228.947479219" Nov 01 20:51:15.462101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.461625 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podStartSLOduration=5.461601643 podStartE2EDuration="5.461601643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:15.431814108 +0000 UTC m=+1229.081865654" watchObservedRunningTime="2026-11-01 20:51:15.461601643 +0000 UTC m=+1229.111653189" Nov 01 20:51:15.610771 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.610738 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:15.645441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.619858 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:15.645441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.619911 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:15.787102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.780729 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:15.787102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:15.780795 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:16.182103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.178490 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.182103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.178563 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:16.183616 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.183580 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.183873 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.183845 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:16.378112 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.377520 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:51:16.402105 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.382618 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" event={"ID":"fd61b1b0-c480-4678-8e78-4e68d4b8b120","Type":"ContainerStarted","Data":"a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05"} Nov 01 20:51:16.466162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.465065 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:16.466162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.465255 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.466162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.465302 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:16.560103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.554068 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:16.593143 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.593105 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.593417 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.593390 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:16.773556 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.773459 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.778178 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.778142 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:16.778394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.773843 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:16.825331 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.825259 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:16.981266 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:16.981226 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:17.093117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.093051 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:17.181795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.181757 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.182504 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.182468 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:17.182685 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.182188 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.182785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.182769 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:17.380159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.379224 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:17.380159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.379616 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:51:17.380159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.379921 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:51:17.487234 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.487197 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.487489 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.487459 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:17.506919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.506850 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podStartSLOduration=7.506828258 podStartE2EDuration="7.506828258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:16.569460554 +0000 UTC m=+1230.219512100" watchObservedRunningTime="2026-11-01 20:51:17.506828258 +0000 UTC m=+1231.156879818" Nov 01 20:51:17.558499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.558453 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:17.559170 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.559145 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:51:17.611684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.611567 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.611684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.611621 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:17.723451 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.723352 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:17.820102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.813700 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:17.820102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.814820 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.820102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.814865 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:17.893231 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.893186 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 20:51:17.955751 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.955716 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:51:18.008161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.981698 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.008161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:17.982126 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:51:18.226101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.211993 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:18.226101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.212085 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:18.226101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.212247 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:18.226101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.212279 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:18.482730 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.424324 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.482730 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.424628 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:51:18.482730 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.424855 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:51:18.557239 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.557190 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.561225 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.561189 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.568126 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.562232 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:18.568126 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.562281 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:18.665237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.665196 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:18.665474 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.665449 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:18.724381 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.724342 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.736179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.736100 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:51:18.758264 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.758235 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:18.758536 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.758498 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:18.818468 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.818419 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.820196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.820169 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:51:18.969320 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.969273 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 20:51:19.002189 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:18.996206 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:19.122477 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.122427 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:19.132205 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.128217 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 20:51:19.207643 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.207607 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.207943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.207906 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:19.226231 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.208151 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.226513 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.226466 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:19.429252 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.429216 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:51:19.464102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.460341 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:19.464102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.460682 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:51:19.491101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.480974 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.496686 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.496643 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:19.568316 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.568273 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:19.664572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.664448 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.664883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.664851 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:19.769923 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.769817 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.770190 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.770157 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:19.820521 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.820473 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:20.009101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.984476 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.010870 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:19.997176 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 01 20:51:20.175905 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.174494 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 01 20:51:20.201099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.198755 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.201099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.198814 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:20.201516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.201490 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.232113 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.201645 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:20.257103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.238295 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.257103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.238887 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:51:20.428293 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.428172 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.502231 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.502174 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.538119 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.505160 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.581554 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.517857 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.585228 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.585164 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:20.590198 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.569187 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.646214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.639671 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.672400 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.672344 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:20.955592 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.955548 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:20.984104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.982195 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.984104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:20.982259 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:21.013102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.012616 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:21.013102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.013062 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 20:51:21.230161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.221236 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:21.230161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.221307 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:21.230161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.221417 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:21.230161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.221441 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:21.230161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.223633 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:21.230161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.224014 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:51:21.243868 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.243823 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:21.433102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.429756 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:21.451410 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.451379 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:51:21.489161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.485213 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:21.489161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.485274 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:21.540102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.531424 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:51:21.613230 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.613195 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:21.613551 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.613518 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:21.641220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.641167 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:51:21.644912 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.644875 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:21.942647 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.942605 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:21.942863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.942667 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:21.969612 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.969571 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:21.969889 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:21.969654 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:22.029101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.027164 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 01 20:51:22.029101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.028551 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:22.177302 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.176428 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:22.177302 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.176483 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:22.177700 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.177617 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 20:51:22.177700 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.177653 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 20:51:22.242458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.238431 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:51:22.242458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.241103 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:22.245716 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.244207 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:22.455526 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.454756 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 01 20:51:22.503167 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.499405 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:22.503167 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.499469 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:22.545813 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.545770 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:22.614789 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.614752 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:22.615089 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.615042 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:22.641658 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.641613 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:22.661038 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.660985 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:22.763387 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.763294 4477 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:51:22.764164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.764134 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:51:22.936732 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.936694 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:51:22.982417 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:22.982356 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:23.036162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.035179 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:23.444781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.439883 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:23.444781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.439995 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:23.444781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.440621 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 01 20:51:23.489159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.465226 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:23.489159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.468240 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:23.489159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.468944 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:23.489159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.469004 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:23.550101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.549675 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:51:23.601104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.598856 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:23.601104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.598925 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:23.642834 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.642792 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:51:23.705007 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.704901 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 01 20:51:23.786403 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.786371 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:51:23.983333 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:23.983219 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:24.025465 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.025401 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:24.185994 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.185852 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:24.185994 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.185920 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:24.292497 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.292409 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:24.294233 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.294212 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:24.307587 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.307546 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:51:24.325149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.310464 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:51:24.408256 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.408205 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:51:24.408527 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.408497 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:24.521101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.507527 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:24.556153 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.551113 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:24.556153 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.551173 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:24.556153 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.552332 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:51:24.646121 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.618038 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:24.647056 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.647003 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:24.661296 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.645455 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:24.707534 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.707492 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:24.994499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.994449 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:24.994684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.994557 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:51:24.994684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.994596 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:51:24.996106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.996064 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:24.997599 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:24.997488 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 20:51:25.012098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.011525 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:51:25.060412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.060381 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:51:25.105129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.105100 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:51:25.174938 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.174897 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:51:25.175126 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.174961 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 20:51:25.462604 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.462572 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:25.462851 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.462818 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:25.592147 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.592113 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 20:51:25.592541 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:25.592177 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 20:51:26.163776 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.163732 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm"] Nov 01 20:51:26.169212 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.169190 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.174787 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.174758 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:26.245437 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.245341 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm"] Nov 01 20:51:26.308721 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.308691 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq92g\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-kube-api-access-bq92g\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.308923 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.308910 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.309062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.309051 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67489e-a66a-454a-acc2-37d86b4cfa84-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.309533 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.309508 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.309745 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.309728 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.309889 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.309877 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.310045 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.310034 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.310231 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.310214 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.310405 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.310390 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412400 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412361 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412623 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412489 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412623 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412527 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412623 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412552 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412819 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412643 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq92g\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-kube-api-access-bq92g\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412819 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412673 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412819 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412783 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67489e-a66a-454a-acc2-37d86b4cfa84-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412987 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412884 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.412987 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.412910 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.413706 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.413665 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.413867 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.413773 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.414065 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.414009 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.414779 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.414752 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67489e-a66a-454a-acc2-37d86b4cfa84-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.415130 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.415062 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.422821 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.417980 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.422821 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.418200 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.465094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.462976 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:51:26.465094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.463026 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 20:51:26.472033 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.471964 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.475758 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.475730 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq92g\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-kube-api-access-bq92g\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.488735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.488538 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:26.522757 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.521412 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:26.598838 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.597338 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:26.632149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.631742 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:26.965218 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.965176 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm"] Nov 01 20:51:26.970650 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:51:26.970607 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e67489e_a66a_454a_acc2_37d86b4cfa84.slice/crio-255feb1bad91daf7154990ec44eadedb82dd137610f1f3cde9666232c7b4bd26 WatchSource:0}: Error finding container 255feb1bad91daf7154990ec44eadedb82dd137610f1f3cde9666232c7b4bd26: Status 404 returned error can't find the container with id 255feb1bad91daf7154990ec44eadedb82dd137610f1f3cde9666232c7b4bd26 Nov 01 20:51:26.972700 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.972666 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:26.972816 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.972739 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:26.972816 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:26.972780 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:51:27.021180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:27.021144 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" event={"ID":"9e67489e-a66a-454a-acc2-37d86b4cfa84","Type":"ContainerStarted","Data":"255feb1bad91daf7154990ec44eadedb82dd137610f1f3cde9666232c7b4bd26"} Nov 01 20:51:27.023112 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:27.023046 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:51:27.024140 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:27.024047 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:51:27.467880 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:27.467847 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:27.468352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:27.468334 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:27.469115 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:27.469096 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 20:51:28.045600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:28.045563 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" event={"ID":"9e67489e-a66a-454a-acc2-37d86b4cfa84","Type":"ContainerStarted","Data":"da04a22f34b40823d2c7aeb31abacfb53ad56d708463a6abf76a242d470d5af4"} Nov 01 20:51:28.164321 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:28.164267 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podStartSLOduration=2.164252515 podStartE2EDuration="2.164252515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:28.1607834 +0000 UTC m=+1241.810834970" watchObservedRunningTime="2026-11-01 20:51:28.164252515 +0000 UTC m=+1241.814304059" Nov 01 20:51:28.489697 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:28.489667 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:28.490780 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:28.490757 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:51:28.490893 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:28.490809 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 20:51:29.448612 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.448579 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8"] Nov 01 20:51:29.452978 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.452955 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.490130 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.490064 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:51:29.490309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.490137 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 20:51:29.546566 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.546526 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8"] Nov 01 20:51:29.553291 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.553260 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:51:29.553493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.553351 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:51:29.569597 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.569566 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.569786 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.569693 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.569846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.569822 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vw6b\" (UniqueName: \"kubernetes.io/projected/0ad0e035-7150-4d70-b0ea-e084f716122b-kube-api-access-8vw6b\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.670994 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.670950 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw6b\" (UniqueName: \"kubernetes.io/projected/0ad0e035-7150-4d70-b0ea-e084f716122b-kube-api-access-8vw6b\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.671275 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.671152 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.671275 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.671243 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.672058 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.672031 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.673652 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.673628 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.736144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.735574 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw6b\" (UniqueName: \"kubernetes.io/projected/0ad0e035-7150-4d70-b0ea-e084f716122b-kube-api-access-8vw6b\") pod \"maas-api-revoke-sub-05674f6b-mjkj8\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:29.771194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:29.771155 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:30.176871 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:30.176842 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8"] Nov 01 20:51:30.178111 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:51:30.178060 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad0e035_7150_4d70_b0ea_e084f716122b.slice/crio-34c39ab9aed86a255eb727819c29b9be5078a47c3877b0fc983a2a11997601be WatchSource:0}: Error finding container 34c39ab9aed86a255eb727819c29b9be5078a47c3877b0fc983a2a11997601be: Status 404 returned error can't find the container with id 34c39ab9aed86a255eb727819c29b9be5078a47c3877b0fc983a2a11997601be Nov 01 20:51:30.490531 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:30.490441 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:51:30.490531 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:30.490493 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 20:51:31.075781 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:31.075741 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" event={"ID":"0ad0e035-7150-4d70-b0ea-e084f716122b","Type":"ContainerStarted","Data":"78adf75e791e492031cac1ecb4e1757b29b9bcef0c57d953bcb969c42dc7ea14"} Nov 01 20:51:31.075995 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:31.075808 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" event={"ID":"0ad0e035-7150-4d70-b0ea-e084f716122b","Type":"ContainerStarted","Data":"34c39ab9aed86a255eb727819c29b9be5078a47c3877b0fc983a2a11997601be"} Nov 01 20:51:31.197909 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:31.197857 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" podStartSLOduration=2.197842653 podStartE2EDuration="2.197842653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:31.194570187 +0000 UTC m=+1244.844621744" watchObservedRunningTime="2026-11-01 20:51:31.197842653 +0000 UTC m=+1244.847894198" Nov 01 20:51:31.490116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:31.489998 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:51:31.490116 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:31.490051 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 20:51:32.084250 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:32.084210 4477 generic.go:356] "Generic (PLEG): container finished" podID="0ad0e035-7150-4d70-b0ea-e084f716122b" containerID="78adf75e791e492031cac1ecb4e1757b29b9bcef0c57d953bcb969c42dc7ea14" exitCode=0 Nov 01 20:51:32.084653 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:32.084297 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" event={"ID":"0ad0e035-7150-4d70-b0ea-e084f716122b","Type":"ContainerDied","Data":"78adf75e791e492031cac1ecb4e1757b29b9bcef0c57d953bcb969c42dc7ea14"} Nov 01 20:51:32.490001 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:32.489904 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:51:32.490231 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:32.490002 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 20:51:33.264102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.264053 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:33.343886 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.343800 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-openshift-ca\") pod \"0ad0e035-7150-4d70-b0ea-e084f716122b\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " Nov 01 20:51:33.344063 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.343896 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad0e035-7150-4d70-b0ea-e084f716122b-kube-api-access-8vw6b\" (UniqueName: \"kubernetes.io/projected/0ad0e035-7150-4d70-b0ea-e084f716122b-kube-api-access-8vw6b\") pod \"0ad0e035-7150-4d70-b0ea-e084f716122b\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " Nov 01 20:51:33.344063 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.344028 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-xks-ca\") pod \"0ad0e035-7150-4d70-b0ea-e084f716122b\" (UID: \"0ad0e035-7150-4d70-b0ea-e084f716122b\") " Nov 01 20:51:33.344755 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.344395 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-openshift-ca" pod "0ad0e035-7150-4d70-b0ea-e084f716122b" (UID: "0ad0e035-7150-4d70-b0ea-e084f716122b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:33.344755 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.344716 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-openshift-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:33.346396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.346363 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad0e035-7150-4d70-b0ea-e084f716122b-kube-api-access-8vw6b" pod "0ad0e035-7150-4d70-b0ea-e084f716122b" (UID: "0ad0e035-7150-4d70-b0ea-e084f716122b"). InnerVolumeSpecName "kube-api-access-8vw6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:33.346598 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.346578 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-xks-ca" pod "0ad0e035-7150-4d70-b0ea-e084f716122b" (UID: "0ad0e035-7150-4d70-b0ea-e084f716122b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:33.445867 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.445820 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vw6b\" (UniqueName: \"kubernetes.io/projected/0ad0e035-7150-4d70-b0ea-e084f716122b-kube-api-access-8vw6b\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:33.445867 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.445866 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0e035-7150-4d70-b0ea-e084f716122b-maas-api-xks-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:33.489769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.489724 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:51:33.489986 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:33.489785 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 20:51:34.105292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:34.105264 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" Nov 01 20:51:34.105553 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:34.105293 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8" event={"ID":"0ad0e035-7150-4d70-b0ea-e084f716122b","Type":"ContainerDied","Data":"34c39ab9aed86a255eb727819c29b9be5078a47c3877b0fc983a2a11997601be"} Nov 01 20:51:34.105687 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:34.105575 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34c39ab9aed86a255eb727819c29b9be5078a47c3877b0fc983a2a11997601be" Nov 01 20:51:34.494807 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:34.494736 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:34.495218 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:34.495200 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:34.495934 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:34.495913 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 20:51:36.555613 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:51:36.555573 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:51:46.891422 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:46.891386 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:51:49.267559 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:49.267519 4477 generic.go:356] "Generic (PLEG): container finished" podID="190501a4-91d5-43b1-96ef-b9f9ca6bde4b" containerID="d84fb203b715f607a17f03c2183c3c52eeba08e0e1088126686338cf916ef225" exitCode=0 Nov 01 20:51:49.279451 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:49.279415 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" event={"ID":"190501a4-91d5-43b1-96ef-b9f9ca6bde4b","Type":"ContainerDied","Data":"d84fb203b715f607a17f03c2183c3c52eeba08e0e1088126686338cf916ef225"} Nov 01 20:51:49.459037 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:49.458994 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:49.544281 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:49.544171 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:49.569234 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:49.569177 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:49.834102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:49.833432 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.026339 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.025168 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.026655 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.026627 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.249289 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.249172 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.329273 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.319155 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 01 20:51:50.499101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.473169 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:51:50.499101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.473956 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.499101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.479230 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:51:50.569101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.565629 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.671105 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.654126 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.933197 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.931223 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.986483 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.986451 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:51:50.988989 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.988967 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:51:50.990063 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:50.990044 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:51:51.040102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.026856 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.061249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.041940 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:51:51.061249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.043905 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.061249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.060317 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:51:51.322618 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.322553 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.376499 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.330280 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:51:51.784102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.752232 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.902151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.786577 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.902151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.819766 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:51:51.902151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.820260 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 01 20:51:51.902151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:51.821057 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:51:52.022226 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.022167 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.073099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.043163 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.085716 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.047222 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.120034 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.085332 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.293808 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.293706 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 20:51:52.401887 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.401839 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 01 20:51:52.760888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.760845 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 01 20:51:52.812191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.812125 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.822521 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.822462 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 01 20:51:52.940261 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.940166 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.999280 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:52.996443 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:51:53.018449 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.018411 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" event={"ID":"190501a4-91d5-43b1-96ef-b9f9ca6bde4b","Type":"ContainerDied","Data":"a16e0edb63f678e05a142df50f52233256851860ac3c74575b004d19ea385e78"} Nov 01 20:51:53.018727 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.018708 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a16e0edb63f678e05a142df50f52233256851860ac3c74575b004d19ea385e78" Nov 01 20:51:53.062290 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.058493 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:51:53.103112 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.093944 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 20:51:53.131159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.131115 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:53.207827 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.207724 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 01 20:51:53.208168 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.208130 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:53.261112 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.249784 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-trusted-ca-bundle\") pod \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " Nov 01 20:51:53.269473 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.269377 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-service-ca-bundle\") pod \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " Nov 01 20:51:53.283771 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.283741 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-kube-api-access-6dqfd\" (UniqueName: \"kubernetes.io/projected/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-kube-api-access-6dqfd\") pod \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " Nov 01 20:51:53.284202 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.284172 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-archives-path\") pod \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\" (UID: \"190501a4-91d5-43b1-96ef-b9f9ca6bde4b\") " Nov 01 20:51:53.287152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.283580 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:53.287414 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.272874 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-service-ca-bundle" pod "190501a4-91d5-43b1-96ef-b9f9ca6bde4b" (UID: "190501a4-91d5-43b1-96ef-b9f9ca6bde4b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:53.287547 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.273346 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-trusted-ca-bundle" pod "190501a4-91d5-43b1-96ef-b9f9ca6bde4b" (UID: "190501a4-91d5-43b1-96ef-b9f9ca6bde4b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:53.362374 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.359633 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-archives-path" pod "190501a4-91d5-43b1-96ef-b9f9ca6bde4b" (UID: "190501a4-91d5-43b1-96ef-b9f9ca6bde4b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:51:53.362374 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.359730 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-kube-api-access-6dqfd" pod "190501a4-91d5-43b1-96ef-b9f9ca6bde4b" (UID: "190501a4-91d5-43b1-96ef-b9f9ca6bde4b"). InnerVolumeSpecName "kube-api-access-6dqfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:53.413152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.392898 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dqfd\" (UniqueName: \"kubernetes.io/projected/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-kube-api-access-6dqfd\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:53.413152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.392930 4477 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-archives-path\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:53.413152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.392945 4477 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-trusted-ca-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:53.413152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.392958 4477 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190501a4-91d5-43b1-96ef-b9f9ca6bde4b-service-ca-bundle\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:53.432141 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.432065 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:53.494692 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.494606 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:51:53.880453 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.880407 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 01 20:51:53.881546 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.881511 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 01 20:51:53.961643 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:53.961604 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:51:54.093170 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.093137 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7304-8d926c"] Nov 01 20:51:54.094414 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.094388 4477 cpu_manager.go:414] "containerMap: removing container" podUID="190501a4-91d5-43b1-96ef-b9f9ca6bde4b" containerName="insights-gathering" Nov 01 20:51:54.098347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.098326 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190501a4-91d5-43b1-96ef-b9f9ca6bde4b" containerName="insights-gathering" Nov 01 20:51:54.098536 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.098522 4477 cpu_manager.go:414] "containerMap: removing container" podUID="0ad0e035-7150-4d70-b0ea-e084f716122b" containerName="revoke-keys" Nov 01 20:51:54.098647 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.098635 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad0e035-7150-4d70-b0ea-e084f716122b" containerName="revoke-keys" Nov 01 20:51:54.135213 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.124538 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.137756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.137720 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.181745 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.181715 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad0e035-7150-4d70-b0ea-e084f716122b" containerName="revoke-keys" Nov 01 20:51:54.181991 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.181975 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="190501a4-91d5-43b1-96ef-b9f9ca6bde4b" containerName="insights-gathering" Nov 01 20:51:54.220735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.220700 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.241308 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.212022 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8rkmx-tkvvw" Nov 01 20:51:54.245800 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.245764 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7304-8d926c"] Nov 01 20:51:54.246120 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.246101 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-8d926c" Nov 01 20:51:54.261580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.261535 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 20:51:54.317552 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.313457 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8mhs\"" Nov 01 20:51:54.547378 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.399339 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwnv\" (UniqueName: \"kubernetes.io/projected/9473d29a-be7b-4af4-8182-335e0b53da3c-kube-api-access-rfwnv\") pod \"test-curl-7304-8d926c\" (UID: \"9473d29a-be7b-4af4-8182-335e0b53da3c\") " pod="opendatahub/test-curl-7304-8d926c" Nov 01 20:51:54.710171 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.691402 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwnv\" (UniqueName: \"kubernetes.io/projected/9473d29a-be7b-4af4-8182-335e0b53da3c-kube-api-access-rfwnv\") pod \"test-curl-7304-8d926c\" (UID: \"9473d29a-be7b-4af4-8182-335e0b53da3c\") " pod="opendatahub/test-curl-7304-8d926c" Nov 01 20:51:54.871897 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.871859 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwnv\" (UniqueName: \"kubernetes.io/projected/9473d29a-be7b-4af4-8182-335e0b53da3c-kube-api-access-rfwnv\") pod \"test-curl-7304-8d926c\" (UID: \"9473d29a-be7b-4af4-8182-335e0b53da3c\") " pod="opendatahub/test-curl-7304-8d926c" Nov 01 20:51:54.890444 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.890324 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.965242 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.965195 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.965999 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.965967 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Nov 01 20:51:54.966280 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.966256 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.966841 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:54.966819 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-8d926c" Nov 01 20:51:55.402862 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.402816 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:55.419351 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.409269 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 01 20:51:55.436140 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.411428 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 20:51:55.577146 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.577018 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:55.577146 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.577033 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:55.588669 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.585959 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:51:55.663796 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.662001 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:51:55.664901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.664200 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:51:55.665728 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.665210 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:51:55.680682 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.680652 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:51:55.718536 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.717568 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:51:55.901257 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.900871 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7304-8d926c"] Nov 01 20:51:55.903803 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:51:55.903773 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9473d29a_be7b_4af4_8182_335e0b53da3c.slice/crio-fb12bb3f7502b0a6f26913df86f5f960106c4964bf65ea71374a5e536ba31aa6 WatchSource:0}: Error finding container fb12bb3f7502b0a6f26913df86f5f960106c4964bf65ea71374a5e536ba31aa6: Status 404 returned error can't find the container with id fb12bb3f7502b0a6f26913df86f5f960106c4964bf65ea71374a5e536ba31aa6 Nov 01 20:51:55.906535 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:55.906506 4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:51:56.492180 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:56.492142 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7304-8d926c" event={"ID":"9473d29a-be7b-4af4-8182-335e0b53da3c","Type":"ContainerStarted","Data":"fb12bb3f7502b0a6f26913df86f5f960106c4964bf65ea71374a5e536ba31aa6"} Nov 01 20:51:59.527531 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:59.527490 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7304-8d926c" event={"ID":"9473d29a-be7b-4af4-8182-335e0b53da3c","Type":"ContainerStarted","Data":"1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932"} Nov 01 20:51:59.650734 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:51:59.650607 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7304-8d926c" podStartSLOduration=3.202880256 podStartE2EDuration="6.65058444s" totalImagesPullingTime="3.447704184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:59.647341159 +0000 UTC m=+1273.297392722" watchObservedRunningTime="2026-11-01 20:51:59.65058444 +0000 UTC m=+1273.300635986" Nov 01 20:52:01.110248 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.110219 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7304-8d926c"] Nov 01 20:52:01.118651 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.118619 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7304-8d926c"] Nov 01 20:52:01.261997 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.261962 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7304-3c520e"] Nov 01 20:52:01.262802 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.262781 4477 cpu_manager.go:414] "containerMap: removing container" podUID="9473d29a-be7b-4af4-8182-335e0b53da3c" containerName="test-curl-7304-8d926c" Nov 01 20:52:01.262939 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.262810 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9473d29a-be7b-4af4-8182-335e0b53da3c" containerName="test-curl-7304-8d926c" Nov 01 20:52:01.262939 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.262896 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="9473d29a-be7b-4af4-8182-335e0b53da3c" containerName="test-curl-7304-8d926c" Nov 01 20:52:01.266460 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.266443 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-3c520e" Nov 01 20:52:01.337064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.333242 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7304-3c520e"] Nov 01 20:52:01.369164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.368216 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w86t\" (UniqueName: \"kubernetes.io/projected/9446049c-9a04-4640-92b8-53cca7644459-kube-api-access-2w86t\") pod \"test-curl-7304-3c520e\" (UID: \"9446049c-9a04-4640-92b8-53cca7644459\") " pod="opendatahub/test-curl-7304-3c520e" Nov 01 20:52:01.469909 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.469876 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w86t\" (UniqueName: \"kubernetes.io/projected/9446049c-9a04-4640-92b8-53cca7644459-kube-api-access-2w86t\") pod \"test-curl-7304-3c520e\" (UID: \"9446049c-9a04-4640-92b8-53cca7644459\") " pod="opendatahub/test-curl-7304-3c520e" Nov 01 20:52:01.526821 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.526782 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w86t\" (UniqueName: \"kubernetes.io/projected/9446049c-9a04-4640-92b8-53cca7644459-kube-api-access-2w86t\") pod \"test-curl-7304-3c520e\" (UID: \"9446049c-9a04-4640-92b8-53cca7644459\") " pod="opendatahub/test-curl-7304-3c520e" Nov 01 20:52:01.547219 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.547165 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7304-8d926c" podUID="9473d29a-be7b-4af4-8182-335e0b53da3c" containerName="test-curl-7304-8d926c" containerID="cri-o://1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932" gracePeriod=2 Nov 01 20:52:01.581973 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.581944 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-3c520e" Nov 01 20:52:01.598308 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:01.598272 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7304-8d926c\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="9473d29a-be7b-4af4-8182-335e0b53da3c" pod="opendatahub/test-curl-7304-8d926c" Nov 01 20:52:01.987124 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:01.987056 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7304-3c520e"] Nov 01 20:52:01.988945 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:52:01.988911 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9446049c_9a04_4640_92b8_53cca7644459.slice/crio-8c06eb53c3d0474efb76fedc8a3e0ff2d337aa525c870d952817f702d967670a WatchSource:0}: Error finding container 8c06eb53c3d0474efb76fedc8a3e0ff2d337aa525c870d952817f702d967670a: Status 404 returned error can't find the container with id 8c06eb53c3d0474efb76fedc8a3e0ff2d337aa525c870d952817f702d967670a Nov 01 20:52:02.556564 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:02.556525 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7304-3c520e" event={"ID":"9446049c-9a04-4640-92b8-53cca7644459","Type":"ContainerStarted","Data":"8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625"} Nov 01 20:52:02.556564 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:02.556569 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7304-3c520e" event={"ID":"9446049c-9a04-4640-92b8-53cca7644459","Type":"ContainerStarted","Data":"8c06eb53c3d0474efb76fedc8a3e0ff2d337aa525c870d952817f702d967670a"} Nov 01 20:52:02.674020 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:02.673947 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7304-3c520e" podStartSLOduration=1.673926467 podStartE2EDuration="1.673926467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:02.667781408 +0000 UTC m=+1276.317832968" watchObservedRunningTime="2026-11-01 20:52:02.673926467 +0000 UTC m=+1276.323978014" Nov 01 20:52:03.875971 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:03.875938 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-8d926c" Nov 01 20:52:04.021024 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.020975 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9473d29a-be7b-4af4-8182-335e0b53da3c-kube-api-access-rfwnv\" (UniqueName: \"kubernetes.io/projected/9473d29a-be7b-4af4-8182-335e0b53da3c-kube-api-access-rfwnv\") pod \"9473d29a-be7b-4af4-8182-335e0b53da3c\" (UID: \"9473d29a-be7b-4af4-8182-335e0b53da3c\") " Nov 01 20:52:04.023368 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.023339 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9473d29a-be7b-4af4-8182-335e0b53da3c-kube-api-access-rfwnv" pod "9473d29a-be7b-4af4-8182-335e0b53da3c" (UID: "9473d29a-be7b-4af4-8182-335e0b53da3c"). InnerVolumeSpecName "kube-api-access-rfwnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:52:04.123401 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.123320 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwnv\" (UniqueName: \"kubernetes.io/projected/9473d29a-be7b-4af4-8182-335e0b53da3c-kube-api-access-rfwnv\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:04.161921 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.161882 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7304-3c520e"] Nov 01 20:52:04.166462 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.166428 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7304-3c520e"] Nov 01 20:52:04.508028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.507941 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9473d29a-be7b-4af4-8182-335e0b53da3c" path="/var/lib/kubelet/pods/9473d29a-be7b-4af4-8182-335e0b53da3c/volumes" Nov 01 20:52:04.573684 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.573651 4477 generic.go:356] "Generic (PLEG): container finished" podID="9473d29a-be7b-4af4-8182-335e0b53da3c" containerID="1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932" exitCode=137 Nov 01 20:52:04.573937 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.573708 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-8d926c" Nov 01 20:52:04.574340 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.574312 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7304-3c520e" podUID="9446049c-9a04-4640-92b8-53cca7644459" containerName="test-curl-7304-3c520e" containerID="cri-o://8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625" gracePeriod=2 Nov 01 20:52:04.574672 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.573735 4477 scope.go:122] "RemoveContainer" containerID="1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932" Nov 01 20:52:04.590435 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.590413 4477 scope.go:122] "RemoveContainer" containerID="1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932" Nov 01 20:52:04.590770 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:04.590741 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932\": container with ID starting with 1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932 not found: ID does not exist" containerID="1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932" Nov 01 20:52:04.590876 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:04.590777 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932"} err="failed to get container status \"1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932\": rpc error: code = NotFound desc = could not find container \"1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932\": container with ID starting with 1b4ca1119d154124c3d7c6f127609c154de0cd7c8b6b605307c25428bfa1e932 not found: ID does not exist" Nov 01 20:52:06.569092 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:06.566234 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:52:06.804589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:06.804558 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-3c520e" Nov 01 20:52:06.968757 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:06.968664 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9446049c-9a04-4640-92b8-53cca7644459-kube-api-access-2w86t\" (UniqueName: \"kubernetes.io/projected/9446049c-9a04-4640-92b8-53cca7644459-kube-api-access-2w86t\") pod \"9446049c-9a04-4640-92b8-53cca7644459\" (UID: \"9446049c-9a04-4640-92b8-53cca7644459\") " Nov 01 20:52:06.971275 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:06.971245 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9446049c-9a04-4640-92b8-53cca7644459-kube-api-access-2w86t" pod "9446049c-9a04-4640-92b8-53cca7644459" (UID: "9446049c-9a04-4640-92b8-53cca7644459"). InnerVolumeSpecName "kube-api-access-2w86t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:52:07.070959 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:07.070928 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w86t\" (UniqueName: \"kubernetes.io/projected/9446049c-9a04-4640-92b8-53cca7644459-kube-api-access-2w86t\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:07.604194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:07.604160 4477 generic.go:356] "Generic (PLEG): container finished" podID="9446049c-9a04-4640-92b8-53cca7644459" containerID="8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625" exitCode=137 Nov 01 20:52:07.604653 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:07.604210 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-3c520e" Nov 01 20:52:07.604653 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:07.604269 4477 scope.go:122] "RemoveContainer" containerID="8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625" Nov 01 20:52:07.616360 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:07.616344 4477 scope.go:122] "RemoveContainer" containerID="8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625" Nov 01 20:52:07.616638 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:07.616612 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625\": container with ID starting with 8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625 not found: ID does not exist" containerID="8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625" Nov 01 20:52:07.616739 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:07.616645 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625"} err="failed to get container status \"8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625\": rpc error: code = NotFound desc = could not find container \"8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625\": container with ID starting with 8cd942c955542e9202af9d503d04bbff229fb81bfbccb25cb99257b41aa25625 not found: ID does not exist" Nov 01 20:52:07.665034 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:07.665001 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7304-3c520e\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="9446049c-9a04-4640-92b8-53cca7644459" pod="opendatahub/test-curl-7304-3c520e" Nov 01 20:52:08.507654 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:08.507619 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9446049c-9a04-4640-92b8-53cca7644459" path="/var/lib/kubelet/pods/9446049c-9a04-4640-92b8-53cca7644459/volumes" Nov 01 20:52:11.231855 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.231819 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b"] Nov 01 20:52:11.232814 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.232790 4477 cpu_manager.go:414] "containerMap: removing container" podUID="9446049c-9a04-4640-92b8-53cca7644459" containerName="test-curl-7304-3c520e" Nov 01 20:52:11.232814 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.232814 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9446049c-9a04-4640-92b8-53cca7644459" containerName="test-curl-7304-3c520e" Nov 01 20:52:11.233036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.233022 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="9446049c-9a04-4640-92b8-53cca7644459" containerName="test-curl-7304-3c520e" Nov 01 20:52:11.238350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.238328 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.281263 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.281230 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c"] Nov 01 20:52:11.288375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.286169 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344113 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344175 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.344347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344229 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42xwb\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-kube-api-access-42xwb\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.344347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344262 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344288 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344320 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344617 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344349 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18017f3b-f320-4349-b79b-3e1b81bfe333-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.344617 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344382 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344617 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344418 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.344617 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344467 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lg7\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-kube-api-access-87lg7\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344617 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344505 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.344617 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344569 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344617 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344593 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb4850c-0729-48d4-95fd-758d5dfdddd3-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344622 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.344975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344738 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.344975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344777 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.344975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344828 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.344975 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.344876 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.373129 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.366992 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b"] Nov 01 20:52:11.373491 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.373226 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c"] Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447247 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447336 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42xwb\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-kube-api-access-42xwb\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447377 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447407 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447445 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447475 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18017f3b-f320-4349-b79b-3e1b81bfe333-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447508 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447546 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447653 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lg7\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-kube-api-access-87lg7\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447697 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447771 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447795 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb4850c-0729-48d4-95fd-758d5dfdddd3-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447825 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.447955 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.448000 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.448045 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.448119 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.450386 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.448192 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.451674 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.450570 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.451674 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.450851 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.451674 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.451116 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.451833 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.451802 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb4850c-0729-48d4-95fd-758d5dfdddd3-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.460787 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.455948 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.460787 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.456028 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.460787 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.456978 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.460787 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.457257 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18017f3b-f320-4349-b79b-3e1b81bfe333-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.460787 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.457500 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.460787 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.457552 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.463719 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.463666 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.475353 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.475326 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.478695 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.478665 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.484448 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.484374 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.515830 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.512571 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.515830 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.514595 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.515830 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.515179 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42xwb\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-kube-api-access-42xwb\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.515830 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.515782 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lg7\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-kube-api-access-87lg7\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:11.572882 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.572834 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:11.605685 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:11.605656 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:12.030318 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.029769 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c"] Nov 01 20:52:12.031209 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:52:12.031181 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb4850c_0729_48d4_95fd_758d5dfdddd3.slice/crio-27f0bc2167806d0e30c354b1572062bfa5d5b2c3b1e1fb13e628e7b11635c543 WatchSource:0}: Error finding container 27f0bc2167806d0e30c354b1572062bfa5d5b2c3b1e1fb13e628e7b11635c543: Status 404 returned error can't find the container with id 27f0bc2167806d0e30c354b1572062bfa5d5b2c3b1e1fb13e628e7b11635c543 Nov 01 20:52:12.032399 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:52:12.032364 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18017f3b_f320_4349_b79b_3e1b81bfe333.slice/crio-34c42dcae1541a2127b8d08121dd328e316ec9b4ac57e48163a52ba4daf88891 WatchSource:0}: Error finding container 34c42dcae1541a2127b8d08121dd328e316ec9b4ac57e48163a52ba4daf88891: Status 404 returned error can't find the container with id 34c42dcae1541a2127b8d08121dd328e316ec9b4ac57e48163a52ba4daf88891 Nov 01 20:52:12.034399 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.034361 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:12.034472 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.034441 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:12.034522 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.034480 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:12.034561 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.034534 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b"] Nov 01 20:52:12.035053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.034978 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:12.035053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.035036 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:12.035233 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.035061 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:12.657168 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.657127 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" event={"ID":"afb4850c-0729-48d4-95fd-758d5dfdddd3","Type":"ContainerStarted","Data":"0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8"} Nov 01 20:52:12.657654 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.657188 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" event={"ID":"afb4850c-0729-48d4-95fd-758d5dfdddd3","Type":"ContainerStarted","Data":"27f0bc2167806d0e30c354b1572062bfa5d5b2c3b1e1fb13e628e7b11635c543"} Nov 01 20:52:12.659012 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.658985 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" event={"ID":"18017f3b-f320-4349-b79b-3e1b81bfe333","Type":"ContainerStarted","Data":"3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b"} Nov 01 20:52:12.659164 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.659025 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" event={"ID":"18017f3b-f320-4349-b79b-3e1b81bfe333","Type":"ContainerStarted","Data":"34c42dcae1541a2127b8d08121dd328e316ec9b4ac57e48163a52ba4daf88891"} Nov 01 20:52:12.782740 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.782687 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" podStartSLOduration=1.782670671 podStartE2EDuration="1.782670671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:12.779029373 +0000 UTC m=+1286.429080922" watchObservedRunningTime="2026-11-01 20:52:12.782670671 +0000 UTC m=+1286.432722216" Nov 01 20:52:12.903939 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:12.903885 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" podStartSLOduration=1.9038709520000001 podStartE2EDuration="1.903870952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:12.899749655 +0000 UTC m=+1286.549801283" watchObservedRunningTime="2026-11-01 20:52:12.903870952 +0000 UTC m=+1286.553922496" Nov 01 20:52:13.590114 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:13.590034 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:13.590338 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:13.590282 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:52:13.590413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:13.590389 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 20:52:13.607834 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:13.607806 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:13.608209 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:13.608120 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:52:13.608412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:13.608385 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 20:52:14.582097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:14.575356 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:52:14.582097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:14.575427 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 20:52:14.606776 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:14.606745 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:52:14.606919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:14.606805 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 20:52:15.577563 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:15.577519 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:52:15.577772 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:15.577604 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 20:52:15.608746 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:15.608688 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:52:15.608746 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:15.608737 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 20:52:16.576101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:16.575513 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:52:16.576101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:16.575577 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 20:52:16.606978 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:16.606943 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:52:16.607175 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:16.606999 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 20:52:17.575099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:17.574390 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:52:17.575099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:17.574448 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 20:52:17.606847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:17.606819 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:52:17.607086 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:17.607048 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 20:52:18.574465 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:18.574426 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:52:18.574656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:18.574497 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 20:52:18.606336 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:18.606298 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:52:18.606739 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:18.606369 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 20:52:19.770445 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:19.770408 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:19.772018 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:19.771996 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:19.774358 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:19.774340 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:52:19.927990 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:19.927955 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:19.928431 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:19.928371 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:19.929507 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:19.929488 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:52:25.411207 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:25.411159 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:25.977103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:25.973299 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:52:26.107101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.094441 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.262155 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.253599 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.262155 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.253711 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 20:52:26.415103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.414186 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.415604 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.415474 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 01 20:52:26.435097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.415850 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:52:26.749161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.736213 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.784145 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.755194 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 01 20:52:26.831119 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.827264 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.908672 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:26.908536 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.035600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.035476 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.067784 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.067754 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:52:27.095573 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.095529 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.147102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.095886 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:52:27.227011 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.222469 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.336252 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.336210 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.371042 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.371003 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:52:27.423215 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.423170 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 01 20:52:27.443993 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.443952 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 20:52:27.445354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.445329 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:52:27.778165 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.755379 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.778165 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.760217 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:52:27.841098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.840434 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.866103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.859184 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:52:27.939103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.914111 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.995103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:27.944512 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:52:28.078352 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.078307 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:52:28.102104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.082216 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:52:28.102104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.082293 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.102104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.084145 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:52:28.102104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.084210 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.104416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.104374 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 20:52:28.237276 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.237237 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.244143 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.237901 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:52:28.370364 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.370260 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.483103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.473587 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.483103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.477389 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.513105 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.503693 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.626733 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.626640 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:52:28.626972 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.626940 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.961165 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.945912 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.995103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:28.970009 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.090103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:29.081938 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 01 20:52:29.165236 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:29.165186 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 20:52:29.203104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:29.168501 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded" Nov 01 20:52:29.310471 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:29.310359 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.473102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:29.432777 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 20:52:29.521730 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:29.485216 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.622019 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:29.621975 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 20:52:29.623021 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:29.622988 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 01 20:52:30.034204 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:29.989184 4477 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.49s" Nov 01 20:52:30.163254 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.033262 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.163857 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.064633 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.164572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.099175 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.188642 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.188588 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.318102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.315434 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.528633 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.528492 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.529149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.528835 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 20:52:30.682537 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.682045 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 20:52:30.910102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:30.909171 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.169103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.168623 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.207229 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.170507 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 01 20:52:31.207229 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.172012 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.207229 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.190560 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.345679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.345620 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.536828 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.536723 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.538427 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.537408 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.703199 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.699268 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 20:52:31.794550 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.794442 4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 01 20:52:31.916873 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.916826 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 01 20:52:31.995924 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:31.994738 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 20:52:32.065887 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.065360 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 20:52:32.138424 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.138396 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 20:52:32.188144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.175526 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 01 20:52:32.243106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.242715 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 20:52:32.311149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.310652 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 20:52:32.358186 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.351781 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 20:52:32.365883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.364829 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 20:52:32.370127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.366271 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 01 20:52:32.413942 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.413915 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 20:52:32.453348 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:32.453323 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 20:52:36.554377 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:36.554336 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:52:45.467670 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.467627 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl"] Nov 01 20:52:45.474744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.474719 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.526701 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.526657 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl"] Nov 01 20:52:45.550660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.550619 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7617fe8-a158-4d77-842f-e56bd94d8a74-openshift-service-ca\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.550834 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.550687 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5wx\" (UniqueName: \"kubernetes.io/projected/c7617fe8-a158-4d77-842f-e56bd94d8a74-kube-api-access-ls5wx\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.550907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.550841 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.550907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.550896 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-metrics-certs\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.570335 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.570306 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f955d283-serving-cert\"" Nov 01 20:52:45.652127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.652095 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7617fe8-a158-4d77-842f-e56bd94d8a74-openshift-service-ca\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.652309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.652144 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5wx\" (UniqueName: \"kubernetes.io/projected/c7617fe8-a158-4d77-842f-e56bd94d8a74-kube-api-access-ls5wx\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.652309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.652209 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.652309 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.652241 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-metrics-certs\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.652486 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:45.652372 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-serving-cert: secret "maas-api-e2e-ait-f955d283-serving-cert" not found Nov 01 20:52:45.652486 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:45.652425 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls podName:c7617fe8-a158-4d77-842f-e56bd94d8a74 nodeName:}" failed. No retries permitted until 2026-11-01 20:52:46.15241023 +0000 UTC m=+1319.802461756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls") pod "maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" (UID: "c7617fe8-a158-4d77-842f-e56bd94d8a74") : secret "maas-api-e2e-ait-f955d283-serving-cert" not found Nov 01 20:52:45.652727 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.652709 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7617fe8-a158-4d77-842f-e56bd94d8a74-openshift-service-ca\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.654601 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.654584 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-metrics-certs\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:45.709768 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:45.709741 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5wx\" (UniqueName: \"kubernetes.io/projected/c7617fe8-a158-4d77-842f-e56bd94d8a74-kube-api-access-ls5wx\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:46.161179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:46.161139 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:46.163857 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:46.163827 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls\") pod \"maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:46.392409 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:46.392373 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:46.747094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:46.746987 4477 scope.go:122] "RemoveContainer" containerID="9503bb938d3c072389d1e44ecd6d485b853686016801aa08000b14bcb1b82410" Nov 01 20:52:46.757274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:46.757253 4477 scope.go:122] "RemoveContainer" containerID="5121710d35a1fe83266efaff25e0f09a7e459a47fecd8422a1a0b485898e11f6" Nov 01 20:52:46.774423 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:46.774397 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl"] Nov 01 20:52:46.775377 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:52:46.775352 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7617fe8_a158_4d77_842f_e56bd94d8a74.slice/crio-7ab64803f84d3944afe4ac7372c2b3fc1e2d8f23ca371eac886994e50f6467e5 WatchSource:0}: Error finding container 7ab64803f84d3944afe4ac7372c2b3fc1e2d8f23ca371eac886994e50f6467e5: Status 404 returned error can't find the container with id 7ab64803f84d3944afe4ac7372c2b3fc1e2d8f23ca371eac886994e50f6467e5 Nov 01 20:52:47.608202 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:47.608158 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" event={"ID":"c7617fe8-a158-4d77-842f-e56bd94d8a74","Type":"ContainerStarted","Data":"7ab64803f84d3944afe4ac7372c2b3fc1e2d8f23ca371eac886994e50f6467e5"} Nov 01 20:52:48.631144 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:48.631107 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" event={"ID":"c7617fe8-a158-4d77-842f-e56bd94d8a74","Type":"ContainerStarted","Data":"a61282b692c5047cf4a378e40773c2a24abd7f2681d7e214fedcf361ab7c277a"} Nov 01 20:52:48.631641 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:48.631311 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:48.745606 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:48.745542 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" podStartSLOduration=2.833132918 podStartE2EDuration="3.745523342s" totalImagesPullingTime="912.390424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:48.744738949 +0000 UTC m=+1322.394790704" watchObservedRunningTime="2026-11-01 20:52:48.745523342 +0000 UTC m=+1322.395574887" Nov 01 20:52:50.361727 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.361686 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4"] Nov 01 20:52:50.366603 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.366572 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.423239 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.423201 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4"] Nov 01 20:52:50.453447 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.453409 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fqv\" (UniqueName: \"kubernetes.io/projected/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-kube-api-access-k6fqv\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.453616 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.453491 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.453686 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.453667 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-openshift-service-ca\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.453738 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.453723 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-metrics-certs\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.466612 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.466583 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a0ef08b4-serving-cert\"" Nov 01 20:52:50.555445 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.555396 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-openshift-service-ca\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.555633 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.555545 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-metrics-certs\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.555758 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.555718 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fqv\" (UniqueName: \"kubernetes.io/projected/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-kube-api-access-k6fqv\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.555836 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.555819 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.556069 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:50.556044 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-serving-cert: secret "maas-api-e2e-ait-a0ef08b4-serving-cert" not found Nov 01 20:52:50.556288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.556262 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-openshift-service-ca\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.556416 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:50.556323 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls podName:59b7cc52-82c8-4a1b-903b-c86e2f45ced7 nodeName:}" failed. No retries permitted until 2026-11-01 20:52:51.056299556 +0000 UTC m=+1324.706351080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls") pod "maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" (UID: "59b7cc52-82c8-4a1b-903b-c86e2f45ced7") : secret "maas-api-e2e-ait-a0ef08b4-serving-cert" not found Nov 01 20:52:50.558303 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.558283 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-metrics-certs\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:50.613049 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:50.612971 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fqv\" (UniqueName: \"kubernetes.io/projected/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-kube-api-access-k6fqv\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:51.065877 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:51.065833 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:51.068650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:51.068621 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls\") pod \"maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:51.280826 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:51.280788 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:51.665622 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:52:51.665592 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b7cc52_82c8_4a1b_903b_c86e2f45ced7.slice/crio-e7b126f1ab5266532819cb2f2318069fe2ac306fd2f80086f70878d7e123528c WatchSource:0}: Error finding container e7b126f1ab5266532819cb2f2318069fe2ac306fd2f80086f70878d7e123528c: Status 404 returned error can't find the container with id e7b126f1ab5266532819cb2f2318069fe2ac306fd2f80086f70878d7e123528c Nov 01 20:52:51.665938 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:51.665673 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4"] Nov 01 20:52:52.688151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:52.687982 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" event={"ID":"59b7cc52-82c8-4a1b-903b-c86e2f45ced7","Type":"ContainerStarted","Data":"8661f306360e969ce927f4f7bc205ad4d479634326f1e1dc3b6adc874cadb7fc"} Nov 01 20:52:52.688151 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:52.688109 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" event={"ID":"59b7cc52-82c8-4a1b-903b-c86e2f45ced7","Type":"ContainerStarted","Data":"e7b126f1ab5266532819cb2f2318069fe2ac306fd2f80086f70878d7e123528c"} Nov 01 20:52:52.688765 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:52.688326 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:52.805247 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:52.805154 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" podStartSLOduration=2.022920002 podStartE2EDuration="2.805139922s" totalImagesPullingTime="782.21992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:52.801211168 +0000 UTC m=+1326.451262725" watchObservedRunningTime="2026-11-01 20:52:52.805139922 +0000 UTC m=+1326.455191503" Nov 01 20:52:54.652190 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:54.652159 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:52:55.251569 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.251533 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq"] Nov 01 20:52:55.256002 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.255979 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.311255 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.311222 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d0d57533-serving-cert\"" Nov 01 20:52:55.315177 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.315145 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq"] Nov 01 20:52:55.335362 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.335325 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8fm\" (UniqueName: \"kubernetes.io/projected/11f0c56e-b73e-4817-adb3-cc7391a319cb-kube-api-access-pd8fm\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.335565 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.335461 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f0c56e-b73e-4817-adb3-cc7391a319cb-openshift-service-ca\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.335565 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.335511 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.335650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.335616 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-metrics-certs\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.435060 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.435028 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk"] Nov 01 20:52:55.436748 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.436724 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8fm\" (UniqueName: \"kubernetes.io/projected/11f0c56e-b73e-4817-adb3-cc7391a319cb-kube-api-access-pd8fm\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.436860 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.436808 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f0c56e-b73e-4817-adb3-cc7391a319cb-openshift-service-ca\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.436860 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.436846 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.436972 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.436921 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-metrics-certs\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.437053 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:55.437029 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-serving-cert: secret "maas-api-e2e-derive-d0d57533-serving-cert" not found Nov 01 20:52:55.437176 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:52:55.437132 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls podName:11f0c56e-b73e-4817-adb3-cc7391a319cb nodeName:}" failed. No retries permitted until 2026-11-01 20:52:55.937109586 +0000 UTC m=+1329.587161132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls") pod "maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" (UID: "11f0c56e-b73e-4817-adb3-cc7391a319cb") : secret "maas-api-e2e-derive-d0d57533-serving-cert" not found Nov 01 20:52:55.437856 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.437812 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f0c56e-b73e-4817-adb3-cc7391a319cb-openshift-service-ca\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.439428 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.439408 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.440012 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.439989 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-metrics-certs\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.495861 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.495825 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cqvhp\"" Nov 01 20:52:55.498769 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.498737 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk"] Nov 01 20:52:55.499847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.499796 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8fm\" (UniqueName: \"kubernetes.io/projected/11f0c56e-b73e-4817-adb3-cc7391a319cb-kube-api-access-pd8fm\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.539349 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.539231 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88f4\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-kube-api-access-s88f4\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.542532 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.542248 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.542532 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.542298 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.542532 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.542402 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.542532 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.542473 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.543364 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.542544 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.543364 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.542580 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.543364 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.542626 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.543364 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.542759 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644408 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644447 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644530 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644588 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644656 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644639 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644670 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644708 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644803 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.644890 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.644856 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88f4\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-kube-api-access-s88f4\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.645370 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.645341 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.645620 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.645600 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.645751 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.645734 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.646249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.646199 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.646249 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.646248 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.648236 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.648153 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.657298 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.657257 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.701905 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.701873 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.702121 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.702006 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88f4\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-kube-api-access-s88f4\") pod \"maas-default-gateway-openshift-default-58c6585447-pfnvk\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.765852 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.765805 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:55.949200 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.949158 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:55.952093 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:55.952037 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls\") pod \"maas-api-e2e-derive-d0d57533-595779b5c-h8vbq\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:56.170836 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.170735 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:56.242860 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.241493 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk"] Nov 01 20:52:56.245662 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:52:56.245631 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bfedee_7a85_49a9_9ae4_f607aca1f8ba.slice/crio-054ed24b6005a2a38c769e964e73eef0740a483460b316cc411508285b96b9d4 WatchSource:0}: Error finding container 054ed24b6005a2a38c769e964e73eef0740a483460b316cc411508285b96b9d4: Status 404 returned error can't find the container with id 054ed24b6005a2a38c769e964e73eef0740a483460b316cc411508285b96b9d4 Nov 01 20:52:56.249647 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.249601 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:56.249745 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.249708 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:56.249786 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.249758 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:52:56.558836 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:52:56.558359 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f0c56e_b73e_4817_adb3_cc7391a319cb.slice/crio-612507bc28fba427c89f6123a787c042329ad8c1a35da17a348e0ed45394dce6 WatchSource:0}: Error finding container 612507bc28fba427c89f6123a787c042329ad8c1a35da17a348e0ed45394dce6: Status 404 returned error can't find the container with id 612507bc28fba427c89f6123a787c042329ad8c1a35da17a348e0ed45394dce6 Nov 01 20:52:56.559828 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.559803 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq"] Nov 01 20:52:56.725705 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.725672 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" event={"ID":"11f0c56e-b73e-4817-adb3-cc7391a319cb","Type":"ContainerStarted","Data":"612507bc28fba427c89f6123a787c042329ad8c1a35da17a348e0ed45394dce6"} Nov 01 20:52:56.727941 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.727914 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" event={"ID":"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba","Type":"ContainerStarted","Data":"bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c"} Nov 01 20:52:56.728113 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.727962 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" event={"ID":"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba","Type":"ContainerStarted","Data":"054ed24b6005a2a38c769e964e73eef0740a483460b316cc411508285b96b9d4"} Nov 01 20:52:56.766790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.766759 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:52:56.846791 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:56.846742 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podStartSLOduration=1.846727226 podStartE2EDuration="1.846727226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:56.842300329 +0000 UTC m=+1330.492351874" watchObservedRunningTime="2026-11-01 20:52:56.846727226 +0000 UTC m=+1330.496778770" Nov 01 20:52:57.756898 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:57.756865 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" event={"ID":"11f0c56e-b73e-4817-adb3-cc7391a319cb","Type":"ContainerStarted","Data":"6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be"} Nov 01 20:52:57.757442 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:57.757113 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:52:57.768542 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:57.768515 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:52:57.768772 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:57.768752 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 20:52:57.893447 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:57.893388 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" podStartSLOduration=1.900708055 podStartE2EDuration="2.893374509s" totalImagesPullingTime="992.666454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:57.891439511 +0000 UTC m=+1331.541491057" watchObservedRunningTime="2026-11-01 20:52:57.893374509 +0000 UTC m=+1331.543426127" Nov 01 20:52:58.702518 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:58.702484 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:52:58.766396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:58.766367 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:52:58.766792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:58.766414 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 20:52:59.766960 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:59.766920 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:52:59.767489 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:52:59.766987 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 20:53:00.767003 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:00.766965 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:53:00.767558 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:00.767022 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 20:53:01.766427 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:01.766389 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:53:01.766637 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:01.766441 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 20:53:02.766643 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:02.766601 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:53:02.767037 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:02.766673 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 20:53:03.766520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:03.766475 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:53:03.766730 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:03.766558 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 20:53:03.768953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:03.768931 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:53:05.287294 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:05.286820 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:53:05.289540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:05.289513 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:53:05.290013 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:05.289984 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 20:53:05.375743 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:05.375627 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:53:06.576204 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:53:06.576166 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:53:10.560344 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.560304 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j"] Nov 01 20:53:10.565985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.565960 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.640917 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.640874 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.641175 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.641058 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.641269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.641194 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.641364 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.641337 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.641469 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.641436 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.641625 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.641547 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.641625 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.641611 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.641750 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.641637 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj8x\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-kube-api-access-gjj8x\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.641750 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.641666 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.692576 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.692547 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j"] Nov 01 20:53:10.751856 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.751815 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.751900 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.751964 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752000 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752053 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752120 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752147 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj8x\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-kube-api-access-gjj8x\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752343 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752176 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752504 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752344 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752504 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752370 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752609 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752518 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.752990 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.752960 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.753135 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.753087 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.753135 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.753091 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.755340 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.755312 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.755764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.755745 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.808446 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.808416 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.808581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.808529 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj8x\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-kube-api-access-gjj8x\") pod \"maas-default-gateway-openshift-default-58c6585447-qzw6j\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:10.883517 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:10.883486 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:11.333441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:11.331622 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j"] Nov 01 20:53:11.343279 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:11.343184 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:53:11.343405 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:11.343288 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:53:11.343405 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:11.343334 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:53:11.874104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:11.874044 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" event={"ID":"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb","Type":"ContainerStarted","Data":"53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c"} Nov 01 20:53:11.874465 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:11.874111 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" event={"ID":"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb","Type":"ContainerStarted","Data":"92a066780555d26283d3960c9bc7435893e49c4715dda8a0de6bbabef4c7d75d"} Nov 01 20:53:11.884481 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:11.884452 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:12.090425 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:12.090369 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podStartSLOduration=2.090355402 podStartE2EDuration="2.090355402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:53:12.03882343 +0000 UTC m=+1345.688874995" watchObservedRunningTime="2026-11-01 20:53:12.090355402 +0000 UTC m=+1345.740406947" Nov 01 20:53:12.885355 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:12.885315 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qzw6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:53:12.885806 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:12.885374 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 20:53:13.884889 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:13.884855 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qzw6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:53:13.885122 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:13.884935 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 20:53:14.884473 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:14.884440 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qzw6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:53:14.884869 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:14.884492 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 20:53:15.884901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:15.884867 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qzw6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:53:15.885354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:15.884919 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 20:53:16.884086 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:16.884047 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qzw6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:53:16.884268 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:16.884124 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 20:53:17.884215 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:17.884184 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qzw6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:53:17.884680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:17.884235 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 20:53:18.884064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:18.884023 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qzw6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:53:18.884260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:18.884124 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 20:53:20.431092 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:20.431022 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:20.431971 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:20.431938 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:20.433252 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:20.433229 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 20:53:36.555498 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:53:36.555451 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:53:40.916006 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:53:40.915970 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:54:06.594818 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:54:06.594719 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:54:33.382153 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:33.382113 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl"] Nov 01 20:54:33.382723 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:33.382436 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" podUID="c7617fe8-a158-4d77-842f-e56bd94d8a74" containerName="maas-api" containerID="cri-o://a61282b692c5047cf4a378e40773c2a24abd7f2681d7e214fedcf361ab7c277a" gracePeriod=30 Nov 01 20:54:33.697206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:33.696829 4477 generic.go:356] "Generic (PLEG): container finished" podID="c7617fe8-a158-4d77-842f-e56bd94d8a74" containerID="a61282b692c5047cf4a378e40773c2a24abd7f2681d7e214fedcf361ab7c277a" exitCode=0 Nov 01 20:54:33.697206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:33.697101 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" event={"ID":"c7617fe8-a158-4d77-842f-e56bd94d8a74","Type":"ContainerDied","Data":"a61282b692c5047cf4a378e40773c2a24abd7f2681d7e214fedcf361ab7c277a"} Nov 01 20:54:34.211787 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.211754 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:54:34.367136 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.367100 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls\") pod \"c7617fe8-a158-4d77-842f-e56bd94d8a74\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " Nov 01 20:54:34.367540 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.367516 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7617fe8-a158-4d77-842f-e56bd94d8a74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7617fe8-a158-4d77-842f-e56bd94d8a74-openshift-service-ca\") pod \"c7617fe8-a158-4d77-842f-e56bd94d8a74\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " Nov 01 20:54:34.368201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.368178 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7617fe8-a158-4d77-842f-e56bd94d8a74-kube-api-access-ls5wx\" (UniqueName: \"kubernetes.io/projected/c7617fe8-a158-4d77-842f-e56bd94d8a74-kube-api-access-ls5wx\") pod \"c7617fe8-a158-4d77-842f-e56bd94d8a74\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " Nov 01 20:54:34.368717 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.368697 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-metrics-certs\") pod \"c7617fe8-a158-4d77-842f-e56bd94d8a74\" (UID: \"c7617fe8-a158-4d77-842f-e56bd94d8a74\") " Nov 01 20:54:34.368994 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.367988 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7617fe8-a158-4d77-842f-e56bd94d8a74-openshift-service-ca" pod "c7617fe8-a158-4d77-842f-e56bd94d8a74" (UID: "c7617fe8-a158-4d77-842f-e56bd94d8a74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:34.369636 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.369610 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7617fe8-a158-4d77-842f-e56bd94d8a74-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:34.408438 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.407825 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls" pod "c7617fe8-a158-4d77-842f-e56bd94d8a74" (UID: "c7617fe8-a158-4d77-842f-e56bd94d8a74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:54:34.408438 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.407852 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-metrics-certs" pod "c7617fe8-a158-4d77-842f-e56bd94d8a74" (UID: "c7617fe8-a158-4d77-842f-e56bd94d8a74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:54:34.408438 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.407947 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7617fe8-a158-4d77-842f-e56bd94d8a74-kube-api-access-ls5wx" pod "c7617fe8-a158-4d77-842f-e56bd94d8a74" (UID: "c7617fe8-a158-4d77-842f-e56bd94d8a74"). InnerVolumeSpecName "kube-api-access-ls5wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:34.471629 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.471553 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:34.471629 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.471580 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls5wx\" (UniqueName: \"kubernetes.io/projected/c7617fe8-a158-4d77-842f-e56bd94d8a74-kube-api-access-ls5wx\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:34.471629 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.471590 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7617fe8-a158-4d77-842f-e56bd94d8a74-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:34.514622 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.514588 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4"] Nov 01 20:54:34.514860 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.514817 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" podUID="59b7cc52-82c8-4a1b-903b-c86e2f45ced7" containerName="maas-api" containerID="cri-o://8661f306360e969ce927f4f7bc205ad4d479634326f1e1dc3b6adc874cadb7fc" gracePeriod=30 Nov 01 20:54:34.708846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.708811 4477 generic.go:356] "Generic (PLEG): container finished" podID="59b7cc52-82c8-4a1b-903b-c86e2f45ced7" containerID="8661f306360e969ce927f4f7bc205ad4d479634326f1e1dc3b6adc874cadb7fc" exitCode=0 Nov 01 20:54:34.709036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.708897 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" event={"ID":"59b7cc52-82c8-4a1b-903b-c86e2f45ced7","Type":"ContainerDied","Data":"8661f306360e969ce927f4f7bc205ad4d479634326f1e1dc3b6adc874cadb7fc"} Nov 01 20:54:34.710943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.710917 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" event={"ID":"c7617fe8-a158-4d77-842f-e56bd94d8a74","Type":"ContainerDied","Data":"7ab64803f84d3944afe4ac7372c2b3fc1e2d8f23ca371eac886994e50f6467e5"} Nov 01 20:54:34.711043 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.710975 4477 scope.go:122] "RemoveContainer" containerID="a61282b692c5047cf4a378e40773c2a24abd7f2681d7e214fedcf361ab7c277a" Nov 01 20:54:34.711159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.711146 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl" Nov 01 20:54:34.885987 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.885947 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl"] Nov 01 20:54:34.902117 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.902068 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f955d283-6b5cf8df57-9h8jl"] Nov 01 20:54:34.905464 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.905438 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:54:34.988988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.987633 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-openshift-service-ca\") pod \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " Nov 01 20:54:34.988988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.987825 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls\") pod \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " Nov 01 20:54:34.988988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.987901 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-kube-api-access-k6fqv\" (UniqueName: \"kubernetes.io/projected/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-kube-api-access-k6fqv\") pod \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " Nov 01 20:54:34.988988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.987989 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-metrics-certs\") pod \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\" (UID: \"59b7cc52-82c8-4a1b-903b-c86e2f45ced7\") " Nov 01 20:54:34.988988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.988333 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-openshift-service-ca" pod "59b7cc52-82c8-4a1b-903b-c86e2f45ced7" (UID: "59b7cc52-82c8-4a1b-903b-c86e2f45ced7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:34.988988 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.988746 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:34.991906 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.991860 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-kube-api-access-k6fqv" pod "59b7cc52-82c8-4a1b-903b-c86e2f45ced7" (UID: "59b7cc52-82c8-4a1b-903b-c86e2f45ced7"). InnerVolumeSpecName "kube-api-access-k6fqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:34.994433 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:34.994415 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-metrics-certs" pod "59b7cc52-82c8-4a1b-903b-c86e2f45ced7" (UID: "59b7cc52-82c8-4a1b-903b-c86e2f45ced7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:54:35.002264 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.002229 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls" pod "59b7cc52-82c8-4a1b-903b-c86e2f45ced7" (UID: "59b7cc52-82c8-4a1b-903b-c86e2f45ced7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:54:35.092529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.091388 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6fqv\" (UniqueName: \"kubernetes.io/projected/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-kube-api-access-k6fqv\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:35.092529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.091428 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:35.092529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.091448 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7cc52-82c8-4a1b-903b-c86e2f45ced7-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:35.753132 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.753096 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq"] Nov 01 20:54:35.754589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.754557 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" podUID="11f0c56e-b73e-4817-adb3-cc7391a319cb" containerName="maas-api" containerID="cri-o://6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be" gracePeriod=30 Nov 01 20:54:35.822965 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.822656 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" event={"ID":"59b7cc52-82c8-4a1b-903b-c86e2f45ced7","Type":"ContainerDied","Data":"e7b126f1ab5266532819cb2f2318069fe2ac306fd2f80086f70878d7e123528c"} Nov 01 20:54:35.823050 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.823002 4477 scope.go:122] "RemoveContainer" containerID="8661f306360e969ce927f4f7bc205ad4d479634326f1e1dc3b6adc874cadb7fc" Nov 01 20:54:35.847547 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:35.823558 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4" Nov 01 20:54:36.108283 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.108240 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4"] Nov 01 20:54:36.116908 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.116862 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0ef08b4-68bfdfd957-dphx4"] Nov 01 20:54:36.322482 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.322450 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:54:36.357208 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.357167 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls\") pod \"11f0c56e-b73e-4817-adb3-cc7391a319cb\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " Nov 01 20:54:36.357385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.357275 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f0c56e-b73e-4817-adb3-cc7391a319cb-kube-api-access-pd8fm\" (UniqueName: \"kubernetes.io/projected/11f0c56e-b73e-4817-adb3-cc7391a319cb-kube-api-access-pd8fm\") pod \"11f0c56e-b73e-4817-adb3-cc7391a319cb\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " Nov 01 20:54:36.357471 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.357458 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-metrics-certs\") pod \"11f0c56e-b73e-4817-adb3-cc7391a319cb\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " Nov 01 20:54:36.358602 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.358465 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f0c56e-b73e-4817-adb3-cc7391a319cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f0c56e-b73e-4817-adb3-cc7391a319cb-openshift-service-ca\") pod \"11f0c56e-b73e-4817-adb3-cc7391a319cb\" (UID: \"11f0c56e-b73e-4817-adb3-cc7391a319cb\") " Nov 01 20:54:36.358977 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.358842 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f0c56e-b73e-4817-adb3-cc7391a319cb-openshift-service-ca" pod "11f0c56e-b73e-4817-adb3-cc7391a319cb" (UID: "11f0c56e-b73e-4817-adb3-cc7391a319cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:36.359644 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.359611 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f0c56e-b73e-4817-adb3-cc7391a319cb-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:36.360409 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.360325 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls" pod "11f0c56e-b73e-4817-adb3-cc7391a319cb" (UID: "11f0c56e-b73e-4817-adb3-cc7391a319cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:54:36.361538 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.361512 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f0c56e-b73e-4817-adb3-cc7391a319cb-kube-api-access-pd8fm" pod "11f0c56e-b73e-4817-adb3-cc7391a319cb" (UID: "11f0c56e-b73e-4817-adb3-cc7391a319cb"). InnerVolumeSpecName "kube-api-access-pd8fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:36.361919 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.361892 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-metrics-certs" pod "11f0c56e-b73e-4817-adb3-cc7391a319cb" (UID: "11f0c56e-b73e-4817-adb3-cc7391a319cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:54:36.476104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.472353 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:36.476104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.472393 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd8fm\" (UniqueName: \"kubernetes.io/projected/11f0c56e-b73e-4817-adb3-cc7391a319cb-kube-api-access-pd8fm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:36.476104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.472407 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f0c56e-b73e-4817-adb3-cc7391a319cb-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:36.618148 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.617458 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b7cc52-82c8-4a1b-903b-c86e2f45ced7" path="/var/lib/kubelet/pods/59b7cc52-82c8-4a1b-903b-c86e2f45ced7/volumes" Nov 01 20:54:36.618374 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.618193 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7617fe8-a158-4d77-842f-e56bd94d8a74" path="/var/lib/kubelet/pods/c7617fe8-a158-4d77-842f-e56bd94d8a74/volumes" Nov 01 20:54:36.637383 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:54:36.637354 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:54:36.807507 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.807476 4477 generic.go:356] "Generic (PLEG): container finished" podID="11f0c56e-b73e-4817-adb3-cc7391a319cb" containerID="6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be" exitCode=0 Nov 01 20:54:36.808347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.807539 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" Nov 01 20:54:36.808347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.807560 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" event={"ID":"11f0c56e-b73e-4817-adb3-cc7391a319cb","Type":"ContainerDied","Data":"6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be"} Nov 01 20:54:36.808347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.807643 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq" event={"ID":"11f0c56e-b73e-4817-adb3-cc7391a319cb","Type":"ContainerDied","Data":"612507bc28fba427c89f6123a787c042329ad8c1a35da17a348e0ed45394dce6"} Nov 01 20:54:36.808347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.807676 4477 scope.go:122] "RemoveContainer" containerID="6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be" Nov 01 20:54:36.852279 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.849092 4477 scope.go:122] "RemoveContainer" containerID="6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be" Nov 01 20:54:36.852279 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:54:36.849637 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be\": container with ID starting with 6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be not found: ID does not exist" containerID="6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be" Nov 01 20:54:36.852279 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.849696 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be"} err="failed to get container status \"6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be\": rpc error: code = NotFound desc = could not find container \"6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be\": container with ID starting with 6e766021826514a6cdebc383d919e96f45a9d4dbab8c24ebf752a9f07a1b94be not found: ID does not exist" Nov 01 20:54:36.989156 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.988236 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq"] Nov 01 20:54:36.994627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:36.993251 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0d57533-595779b5c-h8vbq"] Nov 01 20:54:38.507278 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:38.507248 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f0c56e-b73e-4817-adb3-cc7391a319cb" path="/var/lib/kubelet/pods/11f0c56e-b73e-4817-adb3-cc7391a319cb/volumes" Nov 01 20:54:41.286998 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.286963 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k"] Nov 01 20:54:41.287638 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287622 4477 cpu_manager.go:414] "containerMap: removing container" podUID="59b7cc52-82c8-4a1b-903b-c86e2f45ced7" containerName="maas-api" Nov 01 20:54:41.287682 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287640 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b7cc52-82c8-4a1b-903b-c86e2f45ced7" containerName="maas-api" Nov 01 20:54:41.287682 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287656 4477 cpu_manager.go:414] "containerMap: removing container" podUID="c7617fe8-a158-4d77-842f-e56bd94d8a74" containerName="maas-api" Nov 01 20:54:41.287682 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287661 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7617fe8-a158-4d77-842f-e56bd94d8a74" containerName="maas-api" Nov 01 20:54:41.287682 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287672 4477 cpu_manager.go:414] "containerMap: removing container" podUID="11f0c56e-b73e-4817-adb3-cc7391a319cb" containerName="maas-api" Nov 01 20:54:41.287682 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287679 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f0c56e-b73e-4817-adb3-cc7391a319cb" containerName="maas-api" Nov 01 20:54:41.287834 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287774 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f0c56e-b73e-4817-adb3-cc7391a319cb" containerName="maas-api" Nov 01 20:54:41.287834 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287789 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b7cc52-82c8-4a1b-903b-c86e2f45ced7" containerName="maas-api" Nov 01 20:54:41.287834 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.287798 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7617fe8-a158-4d77-842f-e56bd94d8a74" containerName="maas-api" Nov 01 20:54:41.292744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.292710 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394251 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394206 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394436 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394299 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394436 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394400 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394568 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394449 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394568 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394562 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394681 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394610 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjjz\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-kube-api-access-6hjjz\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394681 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394663 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394782 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394703 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.394782 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.394732 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.405916 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.405886 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k"] Nov 01 20:54:41.496544 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.496508 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.496746 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.496583 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.496746 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.496678 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.496865 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.496826 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.496985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.496964 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.497053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.497022 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjjz\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-kube-api-access-6hjjz\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.497053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.497039 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.497183 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.496970 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.497240 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.497211 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.497292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.497259 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.497292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.497285 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.497847 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.497820 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.498223 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.498196 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.498528 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.498506 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.499851 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.499818 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.499958 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.499934 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.557868 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.557802 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.558061 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.557800 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjjz\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-kube-api-access-6hjjz\") pod \"maas-default-gateway-openshift-default-58c6585447-rfb2k\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:41.607017 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:41.606990 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:42.099023 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:42.098999 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k"] Nov 01 20:54:42.105036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:42.104992 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:54:42.111097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:42.108189 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:54:42.111097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:42.108268 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:54:42.925972 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:42.925556 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" event={"ID":"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d","Type":"ContainerStarted","Data":"77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b"} Nov 01 20:54:42.926634 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:42.926607 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" event={"ID":"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d","Type":"ContainerStarted","Data":"a78345e65a71bfefef463b910aef350e9f154607b20cca92bf203bedd0fa980c"} Nov 01 20:54:43.100926 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:43.100874 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podStartSLOduration=2.100859848 podStartE2EDuration="2.100859848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:43.095450903 +0000 UTC m=+1436.745502468" watchObservedRunningTime="2026-11-01 20:54:43.100859848 +0000 UTC m=+1436.750911394" Nov 01 20:54:43.607283 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:43.607246 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:43.609023 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:43.608999 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rfb2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 20:54:43.609169 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:43.609044 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 20:54:44.612102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:44.608160 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rfb2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 20:54:44.612102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:44.608219 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 20:54:45.608120 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:45.608047 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rfb2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 20:54:45.608314 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:45.608128 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 20:54:46.608001 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:46.607969 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rfb2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 20:54:46.608441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:46.608019 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 20:54:47.607354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:47.607321 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rfb2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 20:54:47.607556 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:47.607371 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 20:54:48.607527 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:48.607489 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rfb2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 20:54:48.607984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:48.607545 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 20:54:49.607672 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:49.607629 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rfb2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 20:54:49.608064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:49.607714 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 20:54:51.110324 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:51.110291 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:51.110836 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:51.110804 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:51.111683 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:51.111665 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 20:54:56.403417 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.403382 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l"] Nov 01 20:54:56.409177 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.409157 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.457935 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.457895 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.458154 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.457958 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9zk\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-kube-api-access-wj9zk\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.458207 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.458164 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.458207 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.458198 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.458358 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.458338 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.458423 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.458391 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.458510 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.458488 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.458579 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.458539 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.458639 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.458588 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.517803 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.517766 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l"] Nov 01 20:54:56.560513 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.560464 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.560722 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.560526 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.560722 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.560605 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.560722 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.560649 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.560722 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.560714 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.560959 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.560750 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.560959 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.560796 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.561114 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.560991 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.561114 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.561049 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9zk\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-kube-api-access-wj9zk\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.561894 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.561867 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.562291 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.562260 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.562413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.562326 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.562809 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.562779 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.563067 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.563042 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.563900 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.563878 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.564963 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.564941 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.617974 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.617941 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.617974 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.617960 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9zk\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-kube-api-access-wj9zk\") pod \"maas-default-gateway-openshift-default-58c6585447-cxh5l\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.724411 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.724317 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:56.964147 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:56.964109 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:54:57.164320 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:57.164292 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l"] Nov 01 20:54:57.165757 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:54:57.165731 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9918869f_25aa_4fbb_ae9f_3881dfd069ca.slice/crio-7db756c3c22ed79598ba61d8785856ba1e5a9a6afd99d17b8ed9a6a824f99359 WatchSource:0}: Error finding container 7db756c3c22ed79598ba61d8785856ba1e5a9a6afd99d17b8ed9a6a824f99359: Status 404 returned error can't find the container with id 7db756c3c22ed79598ba61d8785856ba1e5a9a6afd99d17b8ed9a6a824f99359 Nov 01 20:54:57.168539 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:57.168498 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:54:57.168659 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:57.168573 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:54:57.168659 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:57.168601 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:54:58.154101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:58.152215 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" event={"ID":"9918869f-25aa-4fbb-ae9f-3881dfd069ca","Type":"ContainerStarted","Data":"8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8"} Nov 01 20:54:58.154101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:58.152292 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" event={"ID":"9918869f-25aa-4fbb-ae9f-3881dfd069ca","Type":"ContainerStarted","Data":"7db756c3c22ed79598ba61d8785856ba1e5a9a6afd99d17b8ed9a6a824f99359"} Nov 01 20:54:58.270453 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:58.270398 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podStartSLOduration=2.27038172 podStartE2EDuration="2.27038172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:58.269243786 +0000 UTC m=+1451.919295355" watchObservedRunningTime="2026-11-01 20:54:58.27038172 +0000 UTC m=+1451.920433264" Nov 01 20:54:58.729678 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:58.729645 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:54:58.737243 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:58.737208 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cxh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 20:54:58.737516 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:58.737488 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 20:54:59.725552 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:59.725508 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cxh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 20:54:59.726199 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:54:59.726165 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 20:55:00.452439 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.452404 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5"] Nov 01 20:55:00.456829 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.456800 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.500140 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.500100 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.500342 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.500237 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.500342 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.500269 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.500469 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.500338 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.500469 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.500365 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108661-d8ba-47f9-96ec-cb98a4c2f715-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.500582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.500492 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.500582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.500524 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9fq\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-kube-api-access-cp9fq\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.500685 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.500609 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.501044 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.501017 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.538100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.535630 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-11a069b1-openshift-default-dockercfg-c82c9\"" Nov 01 20:55:00.571783 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.568139 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5"] Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607502 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607591 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607700 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607733 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607802 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607826 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108661-d8ba-47f9-96ec-cb98a4c2f715-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607908 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607939 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9fq\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-kube-api-access-cp9fq\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.607990 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.608930 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.611638 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.611963 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.612823 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.612479 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.613211 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.613184 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108661-d8ba-47f9-96ec-cb98a4c2f715-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.629149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.622698 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.680219 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.680186 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.705944 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.705857 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9fq\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-kube-api-access-cp9fq\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.707132 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.706270 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:00.730127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.725556 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cxh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 20:55:00.730127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.725617 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 20:55:00.773745 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:00.773373 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:01.273402 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:01.273371 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5"] Nov 01 20:55:01.304403 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:55:01.304361 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06108661_d8ba_47f9_96ec_cb98a4c2f715.slice/crio-e4f586455a8f7ccb47abfbd5eddbe1af884210e1959ce3f40b076eab33ef10bc WatchSource:0}: Error finding container e4f586455a8f7ccb47abfbd5eddbe1af884210e1959ce3f40b076eab33ef10bc: Status 404 returned error can't find the container with id e4f586455a8f7ccb47abfbd5eddbe1af884210e1959ce3f40b076eab33ef10bc Nov 01 20:55:01.307828 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:01.307784 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:55:01.307940 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:01.307908 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:55:01.307999 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:01.307952 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:55:01.728534 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:01.728496 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cxh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 20:55:01.728833 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:01.728799 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 20:55:02.039766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.039663 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2"] Nov 01 20:55:02.049194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.049161 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.101351 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.101313 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 20:55:02.101572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.101361 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 20:55:02.101572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.101389 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 20:55:02.101572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.101522 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z57nd\"" Nov 01 20:55:02.152297 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.152245 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2"] Nov 01 20:55:02.160913 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.160177 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.160913 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.160268 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.160913 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.160315 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.160913 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.160395 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72h9c\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-kube-api-access-72h9c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.160913 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.160523 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.160913 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.160686 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.160913 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.160711 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.207103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.206355 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" event={"ID":"06108661-d8ba-47f9-96ec-cb98a4c2f715","Type":"ContainerStarted","Data":"666ef01babc4490ced1e6525ca8b22ffc0111fbd004360adfa84483d1a4d927c"} Nov 01 20:55:02.207103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.206419 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" event={"ID":"06108661-d8ba-47f9-96ec-cb98a4c2f715","Type":"ContainerStarted","Data":"e4f586455a8f7ccb47abfbd5eddbe1af884210e1959ce3f40b076eab33ef10bc"} Nov 01 20:55:02.262397 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.262357 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.262582 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.262488 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72h9c\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-kube-api-access-72h9c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.262641 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.262626 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.262818 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.262805 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.262856 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.262830 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.262935 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.262921 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.263020 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.263005 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.263058 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.263023 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.265902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.265866 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.266108 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.265980 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.266186 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.266141 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.266253 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.266208 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.347174 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.347045 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" podStartSLOduration=2.347014971 podStartE2EDuration="2.347014971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:02.332875003 +0000 UTC m=+1455.982926541" watchObservedRunningTime="2026-11-01 20:55:02.347014971 +0000 UTC m=+1455.997066518" Nov 01 20:55:02.368094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.360950 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72h9c\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-kube-api-access-72h9c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.368094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.361534 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qg2c2\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.368094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.366600 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:02.736159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.729512 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cxh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 20:55:02.736159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.729576 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 20:55:02.774487 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.774452 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:02.781287 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.781257 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:02.847146 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:02.847123 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2"] Nov 01 20:55:03.243272 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:03.243239 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" event={"ID":"d6c53d18-8bfa-4fd1-8622-41c885aee1fd","Type":"ContainerStarted","Data":"0101d0e917ea9a5e046146e55c1e5d43a8f30084f8170dbe3223fe67ee2f06bb"} Nov 01 20:55:03.243715 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:03.243627 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:03.244741 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:03.244720 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:55:03.725166 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:03.725124 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cxh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 20:55:03.725340 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:03.725184 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 20:55:04.724727 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:04.724692 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cxh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 20:55:04.725150 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:04.724762 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 20:55:06.039287 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:06.039253 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:55:06.040763 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:06.040724 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:55:06.041293 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:06.041270 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 20:55:06.561466 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:55:06.561428 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:55:06.961968 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:06.961924 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:55:06.962056 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:06.962015 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:55:07.290958 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:07.290871 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" event={"ID":"d6c53d18-8bfa-4fd1-8622-41c885aee1fd","Type":"ContainerStarted","Data":"ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8"} Nov 01 20:55:07.465825 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:07.465770 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" podStartSLOduration=2.3561898660000002 podStartE2EDuration="6.465755634s" totalImagesPullingTime="4.109565768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:07.461140411 +0000 UTC m=+1461.111191974" watchObservedRunningTime="2026-11-01 20:55:07.465755634 +0000 UTC m=+1461.115807170" Nov 01 20:55:08.312100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:08.304208 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:08.312100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:08.306605 4477 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qg2c2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:55:08.312100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:08.306671 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" podUID="d6c53d18-8bfa-4fd1-8622-41c885aee1fd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:55:09.309110 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:09.309047 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 20:55:36.556291 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:55:36.556204 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:55:57.041457 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.041423 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4"] Nov 01 20:55:57.049452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.049428 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.105003 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.104958 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4"] Nov 01 20:55:57.147175 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.147149 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-11a069b1-serving-cert\"" Nov 01 20:55:57.196855 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.196819 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb4fefff-e5b4-4136-a877-366aba1d0f8e-openshift-service-ca\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.197042 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.196897 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-maas-api-tls\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.197042 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.197017 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw4k6\" (UniqueName: \"kubernetes.io/projected/cb4fefff-e5b4-4136-a877-366aba1d0f8e-kube-api-access-qw4k6\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.197285 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.197267 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-metrics-certs\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.299064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.298957 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-metrics-certs\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.299064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.299036 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb4fefff-e5b4-4136-a877-366aba1d0f8e-openshift-service-ca\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.299354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.299127 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-maas-api-tls\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.299354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.299167 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw4k6\" (UniqueName: \"kubernetes.io/projected/cb4fefff-e5b4-4136-a877-366aba1d0f8e-kube-api-access-qw4k6\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.299772 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.299747 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb4fefff-e5b4-4136-a877-366aba1d0f8e-openshift-service-ca\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.301831 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.301809 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-metrics-certs\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.301951 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.301854 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-maas-api-tls\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.356529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.356497 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw4k6\" (UniqueName: \"kubernetes.io/projected/cb4fefff-e5b4-4136-a877-366aba1d0f8e-kube-api-access-qw4k6\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-mfwv4\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.365415 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.365387 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:57.751275 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:57.751243 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4"] Nov 01 20:55:57.751418 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:55:57.751394 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4fefff_e5b4_4136_a877_366aba1d0f8e.slice/crio-50353a6b9c12f7eff3d2e71b2e5d9141cce0d4f3c6afb8d9d7f9740877c2bf31 WatchSource:0}: Error finding container 50353a6b9c12f7eff3d2e71b2e5d9141cce0d4f3c6afb8d9d7f9740877c2bf31: Status 404 returned error can't find the container with id 50353a6b9c12f7eff3d2e71b2e5d9141cce0d4f3c6afb8d9d7f9740877c2bf31 Nov 01 20:55:58.727700 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:58.727668 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" event={"ID":"cb4fefff-e5b4-4136-a877-366aba1d0f8e","Type":"ContainerStarted","Data":"50353a6b9c12f7eff3d2e71b2e5d9141cce0d4f3c6afb8d9d7f9740877c2bf31"} Nov 01 20:55:59.736437 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:59.736391 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" event={"ID":"cb4fefff-e5b4-4136-a877-366aba1d0f8e","Type":"ContainerStarted","Data":"aa63ebcc363f33ed7c86f81976fd6eccc7649384fdbef3b55882251e377fbf38"} Nov 01 20:55:59.736920 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:59.736467 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:55:59.854977 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:55:59.854920 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" podStartSLOduration=2.915836225 podStartE2EDuration="3.854905194s" totalImagesPullingTime="939.068969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:59.850754984 +0000 UTC m=+1513.500806523" watchObservedRunningTime="2026-11-01 20:55:59.854905194 +0000 UTC m=+1513.504956771" Nov 01 20:56:05.750377 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:05.750338 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 20:56:06.553799 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:56:06.553765 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:56:12.486623 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:12.486583 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth"] Nov 01 20:56:12.487121 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:12.486892 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" containerID="cri-o://ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae" gracePeriod=30 Nov 01 20:56:17.064195 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.064155 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:56:17.797670 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.797644 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:56:17.892556 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.892525 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-token\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-token\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.892743 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.892674 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-podinfo\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.892743 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.892737 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-socket\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.892861 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.892766 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-credential-socket\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.892861 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.892826 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326305c7-129f-4b07-9c0d-6fb1aff5167b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326305c7-129f-4b07-9c0d-6fb1aff5167b-istiod-ca-cert\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.892970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.892872 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-certs\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.892970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.892909 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-kube-api-access-4chs5\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-kube-api-access-4chs5\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.892970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.892939 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-data\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.893162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.893006 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-envoy\") pod \"326305c7-129f-4b07-9c0d-6fb1aff5167b\" (UID: \"326305c7-129f-4b07-9c0d-6fb1aff5167b\") " Nov 01 20:56:17.893162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.893062 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-credential-socket" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:17.893162 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.893095 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-socket" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:17.893396 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.893349 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326305c7-129f-4b07-9c0d-6fb1aff5167b-istiod-ca-cert" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:17.893650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.893405 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-data" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:17.894163 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.893964 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326305c7-129f-4b07-9c0d-6fb1aff5167b-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:17.894163 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.893993 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:17.894163 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.894010 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:17.894163 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.894026 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:17.895600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.895521 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-podinfo" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:56:17.895888 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.895839 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-envoy" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:17.896064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.895953 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-kube-api-access-4chs5" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "kube-api-access-4chs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:17.896064 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.896010 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-token" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:17.896271 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.896056 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-certs" pod "326305c7-129f-4b07-9c0d-6fb1aff5167b" (UID: "326305c7-129f-4b07-9c0d-6fb1aff5167b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:17.914821 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.914791 4477 generic.go:356] "Generic (PLEG): container finished" podID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerID="ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae" exitCode=0 Nov 01 20:56:17.914931 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.914844 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" Nov 01 20:56:17.914931 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.914870 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" event={"ID":"326305c7-129f-4b07-9c0d-6fb1aff5167b","Type":"ContainerDied","Data":"ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae"} Nov 01 20:56:17.915051 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.914969 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth" event={"ID":"326305c7-129f-4b07-9c0d-6fb1aff5167b","Type":"ContainerDied","Data":"db7198a3172cbc301197b97404de2a67df9d36120dfc02d68a2f3b1003ff4340"} Nov 01 20:56:17.915051 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.914998 4477 scope.go:122] "RemoveContainer" containerID="ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae" Nov 01 20:56:17.933660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.933638 4477 scope.go:122] "RemoveContainer" containerID="ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae" Nov 01 20:56:17.934013 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:56:17.933994 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae\": container with ID starting with ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae not found: ID does not exist" containerID="ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae" Nov 01 20:56:17.934094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.934033 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae"} err="failed to get container status \"ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae\": rpc error: code = NotFound desc = could not find container \"ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae\": container with ID starting with ecc0c290e15781d33fcb4c1b6c39c0848a0c3f7520a5fc1bea10c43b2481fcae not found: ID does not exist" Nov 01 20:56:17.996345 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.996312 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:17.996345 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.996341 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:17.996345 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.996351 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:17.996631 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.996359 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4chs5\" (UniqueName: \"kubernetes.io/projected/326305c7-129f-4b07-9c0d-6fb1aff5167b-kube-api-access-4chs5\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:17.996631 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:17.996369 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326305c7-129f-4b07-9c0d-6fb1aff5167b-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.080165 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:18.080136 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth"] Nov 01 20:56:18.084642 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:18.084613 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-nrzth"] Nov 01 20:56:18.508594 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:18.508518 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" path="/var/lib/kubelet/pods/326305c7-129f-4b07-9c0d-6fb1aff5167b/volumes" Nov 01 20:56:21.100097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:21.099159 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:21.142101 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:21.141121 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:21.196098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:21.195633 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:21.349111 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:21.348187 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:56:21.349111 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:21.348265 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:21.499059 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:21.498212 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:21.582106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:21.575210 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:22.093240 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:22.093198 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cxh5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:56:22.093463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:22.093280 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:22.159274 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:22.159237 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rfb2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:56:22.159883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:22.159847 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:29.178701 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.178603 4477 generic.go:356] "Generic (PLEG): container finished" podID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerID="666ef01babc4490ced1e6525ca8b22ffc0111fbd004360adfa84483d1a4d927c" exitCode=0 Nov 01 20:56:29.179271 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.178687 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" event={"ID":"06108661-d8ba-47f9-96ec-cb98a4c2f715","Type":"ContainerDied","Data":"666ef01babc4490ced1e6525ca8b22ffc0111fbd004360adfa84483d1a4d927c"} Nov 01 20:56:29.179271 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.179239 4477 scope.go:122] "RemoveContainer" containerID="666ef01babc4490ced1e6525ca8b22ffc0111fbd004360adfa84483d1a4d927c" Nov 01 20:56:29.181293 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.181260 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:56:29.181385 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.181355 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:56:29.181423 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.181402 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:56:29.634519 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.634469 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx"] Nov 01 20:56:29.635292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.635264 4477 cpu_manager.go:414] "containerMap: removing container" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" Nov 01 20:56:29.635292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.635291 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" Nov 01 20:56:29.635510 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.635489 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="326305c7-129f-4b07-9c0d-6fb1aff5167b" containerName="istio-proxy" Nov 01 20:56:29.639840 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.639819 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.693012 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.692977 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a5ab4417-openshift-default-dockercfg-4tcv5\"" Nov 01 20:56:29.701432 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.701394 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx"] Nov 01 20:56:29.711859 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.711828 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbk2l\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-kube-api-access-dbk2l\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.712055 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.711908 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.712161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.712059 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.712161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.712145 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.712339 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.712185 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.712339 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.712314 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.712449 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.712347 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.712504 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.712465 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.712504 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.712498 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.814558 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.814521 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.814775 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.814595 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.814775 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.814621 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.814775 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.814651 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.814775 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.814732 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.814775 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.814762 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.815062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.814830 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.815062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.814860 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.815062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.815005 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.815062 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.815044 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbk2l\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-kube-api-access-dbk2l\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.815323 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.815045 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.815382 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.815322 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.815639 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.815605 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.816254 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.816170 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.819058 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.819032 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.822534 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.822514 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.872328 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.872296 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.872328 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.872306 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbk2l\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-kube-api-access-dbk2l\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:29.954774 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:29.954694 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:30.191580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:30.191548 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" event={"ID":"06108661-d8ba-47f9-96ec-cb98a4c2f715","Type":"ContainerStarted","Data":"a67b1596a6958ea8d5717ca774c9c04436b7ca1f892a9c36563e2e9f4522c40d"} Nov 01 20:56:30.461241 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:56:30.461210 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a3a587_3e33_4355_b2e4_ae18cd8291da.slice/crio-4ed0cea2329e6e39d42e9708fcc1a824a61635d9438331b2127040f928ed5eca WatchSource:0}: Error finding container 4ed0cea2329e6e39d42e9708fcc1a824a61635d9438331b2127040f928ed5eca: Status 404 returned error can't find the container with id 4ed0cea2329e6e39d42e9708fcc1a824a61635d9438331b2127040f928ed5eca Nov 01 20:56:30.461241 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:30.461231 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx"] Nov 01 20:56:30.464149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:30.464113 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:56:30.464256 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:30.464187 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:56:30.464256 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:30.464217 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:56:30.777156 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:30.774786 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:56:30.787127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:30.781165 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 20:56:30.787127 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:30.781215 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 20:56:31.254099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:31.237286 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" event={"ID":"a3a3a587-3e33-4355-b2e4-ae18cd8291da","Type":"ContainerStarted","Data":"4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e"} Nov 01 20:56:31.254099 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:31.237352 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" event={"ID":"a3a3a587-3e33-4355-b2e4-ae18cd8291da","Type":"ContainerStarted","Data":"4ed0cea2329e6e39d42e9708fcc1a824a61635d9438331b2127040f928ed5eca"} Nov 01 20:56:31.355413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:31.355348 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podStartSLOduration=2.355327525 podStartE2EDuration="2.355327525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:31.34931036 +0000 UTC m=+1544.999361928" watchObservedRunningTime="2026-11-01 20:56:31.355327525 +0000 UTC m=+1545.005379071" Nov 01 20:56:31.777538 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:31.777504 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 20:56:31.777817 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:31.777790 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 20:56:31.958327 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:31.958292 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:31.958766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:31.958744 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 20:56:31.958940 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:31.958917 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 20:56:32.774620 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:32.774578 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 20:56:32.775145 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:32.774636 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 20:56:32.967805 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:32.962528 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 20:56:32.967805 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:32.962587 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 20:56:33.247100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:33.246329 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:56:33.774662 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:33.774620 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 20:56:33.775087 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:33.774671 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 20:56:33.958103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:33.955479 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 20:56:33.958103 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:33.955537 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 20:56:34.070883 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:34.070851 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8"] Nov 01 20:56:34.074953 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:34.074929 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05674f6b-mjkj8"] Nov 01 20:56:34.506322 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:34.506243 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad0e035-7150-4d70-b0ea-e084f716122b" path="/var/lib/kubelet/pods/0ad0e035-7150-4d70-b0ea-e084f716122b/volumes" Nov 01 20:56:34.774637 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:34.774537 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 20:56:34.774637 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:34.774589 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 20:56:34.960102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:34.956688 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 20:56:34.960102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:34.956759 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 20:56:35.779011 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:35.778975 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:56:35.779991 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:35.779969 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 20:56:35.955350 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:35.955317 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 20:56:35.955515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:35.955376 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 20:56:36.571367 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:56:36.571323 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:56:36.955323 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:36.955214 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 20:56:36.955323 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:36.955307 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 20:56:37.961843 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:37.961813 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:37.962361 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:37.962297 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:37.963155 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:37.963133 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:56:50.576775 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:50.576727 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:50.594108 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:50.580278 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:51.358823 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:51.358204 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:56:51.358823 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:51.358286 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:58.609392 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.609312 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r"] Nov 01 20:56:58.614819 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.614796 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.668658 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.668623 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r"] Nov 01 20:56:58.708325 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.708288 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb31a6c3-1a59-4b34-b906-d1a335333566-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.708325 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.708326 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.708591 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.708490 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.708660 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.708604 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxvs\" (UniqueName: \"kubernetes.io/projected/bb31a6c3-1a59-4b34-b906-d1a335333566-kube-api-access-wkxvs\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.712320 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.712286 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-dce296-serving-cert\"" Nov 01 20:56:58.810207 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.810172 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.810375 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.810275 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxvs\" (UniqueName: \"kubernetes.io/projected/bb31a6c3-1a59-4b34-b906-d1a335333566-kube-api-access-wkxvs\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.810417 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.810403 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb31a6c3-1a59-4b34-b906-d1a335333566-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.810453 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.810422 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.810605 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:56:58.810592 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-serving-cert: secret "maas-api-e2e-shared-a-w1-dce296-serving-cert" not found Nov 01 20:56:58.810668 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:56:58.810660 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls podName:bb31a6c3-1a59-4b34-b906-d1a335333566 nodeName:}" failed. No retries permitted until 2026-11-01 20:56:59.310645094 +0000 UTC m=+1572.960696618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls") pod "maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" (UID: "bb31a6c3-1a59-4b34-b906-d1a335333566") : secret "maas-api-e2e-shared-a-w1-dce296-serving-cert" not found Nov 01 20:56:58.811033 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.811015 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb31a6c3-1a59-4b34-b906-d1a335333566-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.812900 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.812876 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:58.873161 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:58.873021 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxvs\" (UniqueName: \"kubernetes.io/projected/bb31a6c3-1a59-4b34-b906-d1a335333566-kube-api-access-wkxvs\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:59.317406 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:59.317370 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:59.320049 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:59.320025 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:59.529206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:59.529174 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:56:59.911981 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:59.911950 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r"] Nov 01 20:56:59.912393 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:56:59.912174 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb31a6c3_1a59_4b34_b906_d1a335333566.slice/crio-d7a96a48a1e78190864ed9662593f1015ed996e18400288129b6f4391ae381f6 WatchSource:0}: Error finding container d7a96a48a1e78190864ed9662593f1015ed996e18400288129b6f4391ae381f6: Status 404 returned error can't find the container with id d7a96a48a1e78190864ed9662593f1015ed996e18400288129b6f4391ae381f6 Nov 01 20:56:59.913863 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:56:59.913844 4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:57:00.810985 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:00.810954 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" event={"ID":"bb31a6c3-1a59-4b34-b906-d1a335333566","Type":"ContainerStarted","Data":"d7a96a48a1e78190864ed9662593f1015ed996e18400288129b6f4391ae381f6"} Nov 01 20:57:01.851807 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:01.851765 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" event={"ID":"bb31a6c3-1a59-4b34-b906-d1a335333566","Type":"ContainerStarted","Data":"4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28"} Nov 01 20:57:01.853661 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:01.853538 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:57:01.983147 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:01.978989 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" podStartSLOduration=3.123737678 podStartE2EDuration="3.978964605s" totalImagesPullingTime="855.226927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:01.966505839 +0000 UTC m=+1575.616557406" watchObservedRunningTime="2026-11-01 20:57:01.978964605 +0000 UTC m=+1575.629016151" Nov 01 20:57:06.555523 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:57:06.555483 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:57:08.873868 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:08.873838 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 20:57:14.981892 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:14.981856 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v"] Nov 01 20:57:14.986455 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:14.986436 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.087853 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.087820 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v"] Nov 01 20:57:15.088179 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088155 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.088294 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088229 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.088354 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088335 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbcvb\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-kube-api-access-rbcvb\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.088495 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088469 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.088630 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088537 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.088630 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088567 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.088630 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088600 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.088873 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088640 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.088873 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.088718 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.190579 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190546 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.190764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190663 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.190764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190714 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.190764 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190738 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbcvb\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-kube-api-access-rbcvb\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.190943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190832 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.190943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190876 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.190943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190899 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.190943 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190934 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.191190 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.190972 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.191190 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.191065 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.191190 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.191127 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.191422 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.191401 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.191475 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.191433 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.191520 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.191499 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.193481 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.193454 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.193709 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.193687 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.247631 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.247544 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.247761 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.247650 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbcvb\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-kube-api-access-rbcvb\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.301041 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.301016 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:15.739825 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.739798 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v"] Nov 01 20:57:15.741545 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:57:15.741514 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0149e00e_0b29_4be7_bd28_0dbdf08a338a.slice/crio-9e0c411c7ec8a6fb304fa9f9f1075597cf61a5dcbd3767a0ab1631cf4aced22c WatchSource:0}: Error finding container 9e0c411c7ec8a6fb304fa9f9f1075597cf61a5dcbd3767a0ab1631cf4aced22c: Status 404 returned error can't find the container with id 9e0c411c7ec8a6fb304fa9f9f1075597cf61a5dcbd3767a0ab1631cf4aced22c Nov 01 20:57:15.744760 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.744724 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:57:15.744848 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.744812 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:57:15.744909 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.744859 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 20:57:15.971005 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.970966 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" event={"ID":"0149e00e-0b29-4be7-bd28-0dbdf08a338a","Type":"ContainerStarted","Data":"5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee"} Nov 01 20:57:15.971243 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:15.971043 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" event={"ID":"0149e00e-0b29-4be7-bd28-0dbdf08a338a","Type":"ContainerStarted","Data":"9e0c411c7ec8a6fb304fa9f9f1075597cf61a5dcbd3767a0ab1631cf4aced22c"} Nov 01 20:57:16.091008 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:16.090941 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podStartSLOduration=2.090923006 podStartE2EDuration="2.090923006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:16.087003027 +0000 UTC m=+1589.737054597" watchObservedRunningTime="2026-11-01 20:57:16.090923006 +0000 UTC m=+1589.740974552" Nov 01 20:57:16.301711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:16.301678 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:16.302963 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:16.302934 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 20:57:16.303111 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:16.302993 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 20:57:17.360097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:17.355438 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 20:57:17.360097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:17.355511 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 20:57:18.302098 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:18.302041 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 20:57:18.302257 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:18.302125 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 20:57:19.301945 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:19.301908 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 20:57:19.302421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:19.301969 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 20:57:20.301696 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:20.301660 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 20:57:20.301870 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:20.301721 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 20:57:21.302036 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:21.301997 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 20:57:21.302466 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:21.302068 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 20:57:22.308340 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:22.308306 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 20:57:22.308977 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:22.308933 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 20:57:23.150902 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:23.150862 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:57:23.653426 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:23.653372 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:23.655937 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:23.655902 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:23.658196 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:23.658170 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:57:36.566410 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:57:36.566367 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:57:43.445441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.445407 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt"] Nov 01 20:57:43.445887 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.445628 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" containerID="cri-o://ce01ca70074e110490301b0840eaeb5ee10c170fcb4780ffbd37f620316b5519" gracePeriod=30 Nov 01 20:57:43.495592 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.495555 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k"] Nov 01 20:57:43.495825 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.495796 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" containerID="cri-o://2661f25d453e0c232205a887303de3008a15aec20511a0fb45996db6bb02fcbb" gracePeriod=30 Nov 01 20:57:43.500427 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.500400 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9"] Nov 01 20:57:43.500618 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.500585 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" containerID="cri-o://3122c958df5be523e7f158a94de52086f2e3c274bbbebef2d3f00c501a385dbd" gracePeriod=30 Nov 01 20:57:43.889618 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.889580 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth"] Nov 01 20:57:43.889822 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.889800 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" containerID="cri-o://c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873" gracePeriod=30 Nov 01 20:57:43.894332 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.894304 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b"] Nov 01 20:57:43.894515 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.894499 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" containerID="cri-o://3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b" gracePeriod=30 Nov 01 20:57:43.898688 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.898660 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c"] Nov 01 20:57:43.898881 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:43.898860 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" containerID="cri-o://0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8" gracePeriod=30 Nov 01 20:57:46.927691 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:46.927652 4477 scope.go:122] "RemoveContainer" containerID="78adf75e791e492031cac1ecb4e1757b29b9bcef0c57d953bcb969c42dc7ea14" Nov 01 20:57:48.234268 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.234224 4477 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:57:48.234710 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.234311 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:57:48.665427 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.665389 4477 generic.go:356] "Generic (PLEG): container finished" podID="49c85304-e51f-434e-a787-bb4315eabe39" containerID="2661f25d453e0c232205a887303de3008a15aec20511a0fb45996db6bb02fcbb" exitCode=0 Nov 01 20:57:48.665616 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.665533 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" event={"ID":"49c85304-e51f-434e-a787-bb4315eabe39","Type":"ContainerDied","Data":"2661f25d453e0c232205a887303de3008a15aec20511a0fb45996db6bb02fcbb"} Nov 01 20:57:48.667627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.667600 4477 generic.go:356] "Generic (PLEG): container finished" podID="921a5139-633b-492a-8c56-4edeedce6f97" containerID="3122c958df5be523e7f158a94de52086f2e3c274bbbebef2d3f00c501a385dbd" exitCode=0 Nov 01 20:57:48.667795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.667776 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" event={"ID":"921a5139-633b-492a-8c56-4edeedce6f97","Type":"ContainerDied","Data":"3122c958df5be523e7f158a94de52086f2e3c274bbbebef2d3f00c501a385dbd"} Nov 01 20:57:48.669583 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.669502 4477 generic.go:356] "Generic (PLEG): container finished" podID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerID="ce01ca70074e110490301b0840eaeb5ee10c170fcb4780ffbd37f620316b5519" exitCode=0 Nov 01 20:57:48.669679 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.669584 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" event={"ID":"f3cc78a3-f034-4873-9e85-0481a94d6818","Type":"ContainerDied","Data":"ce01ca70074e110490301b0840eaeb5ee10c170fcb4780ffbd37f620316b5519"} Nov 01 20:57:48.807007 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.806981 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:57:48.859995 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.851187 4477 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:57:48.859995 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.851256 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:57:48.927260 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.927235 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:57:48.936997 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.936963 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-token\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-token\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.937182 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937108 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-envoy\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.937288 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937180 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-podinfo\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.937359 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937313 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-socket\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.937411 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937361 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-data\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.937411 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937397 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-kube-api-access-lk8tv\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-kube-api-access-lk8tv\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937569 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-certs\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937600 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-socket" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937618 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cc78a3-f034-4873-9e85-0481a94d6818-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cc78a3-f034-4873-9e85-0481a94d6818-istiod-ca-cert\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937651 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-credential-socket\") pod \"f3cc78a3-f034-4873-9e85-0481a94d6818\" (UID: \"f3cc78a3-f034-4873-9e85-0481a94d6818\") " Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937730 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-data" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.937828 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-certs" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.938055 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-credential-socket" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.938277 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cc78a3-f034-4873-9e85-0481a94d6818-istiod-ca-cert" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.938688 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.938789 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cc78a3-f034-4873-9e85-0481a94d6818-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.938805 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.938818 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.938831 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:48.941463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.941048 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-podinfo" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:48.942286 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.941542 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-kube-api-access-lk8tv" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "kube-api-access-lk8tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:48.942286 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.941601 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-envoy" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:48.942286 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:48.942050 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-token" pod "f3cc78a3-f034-4873-9e85-0481a94d6818" (UID: "f3cc78a3-f034-4873-9e85-0481a94d6818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040183 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-envoy\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040226 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-credential-socket\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040296 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-istio-token\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-istio-token\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040427 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-data\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040560 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-kube-api-access-mqbw8\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-kube-api-access-mqbw8\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040627 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c85304-e51f-434e-a787-bb4315eabe39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c85304-e51f-434e-a787-bb4315eabe39-istiod-ca-cert\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040654 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49c85304-e51f-434e-a787-bb4315eabe39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c85304-e51f-434e-a787-bb4315eabe39-istio-podinfo\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040715 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-certs\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.040847 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-socket\") pod \"49c85304-e51f-434e-a787-bb4315eabe39\" (UID: \"49c85304-e51f-434e-a787-bb4315eabe39\") " Nov 01 20:57:49.051878 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.051571 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c85304-e51f-434e-a787-bb4315eabe39-istiod-ca-cert" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:49.052747 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.051905 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-data" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.052747 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.052232 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-certs" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.052747 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.052393 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-credential-socket" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.052909 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.052877 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-socket" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053629 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053660 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053677 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053691 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053715 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cc78a3-f034-4873-9e85-0481a94d6818-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053730 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c85304-e51f-434e-a787-bb4315eabe39-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053744 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053759 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk8tv\" (UniqueName: \"kubernetes.io/projected/f3cc78a3-f034-4873-9e85-0481a94d6818-kube-api-access-lk8tv\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.057142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.053773 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.075287 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.075195 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:57:49.078549 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.078460 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-istio-token" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.078549 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.078502 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-envoy" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.081690 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.081647 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49c85304-e51f-434e-a787-bb4315eabe39-istio-podinfo" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:49.081690 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.081685 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-kube-api-access-mqbw8" pod "49c85304-e51f-434e-a787-bb4315eabe39" (UID: "49c85304-e51f-434e-a787-bb4315eabe39"). InnerVolumeSpecName "kube-api-access-mqbw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156221 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-istio-token\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-istio-token\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156311 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-envoy\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156372 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921a5139-633b-492a-8c56-4edeedce6f97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a5139-633b-492a-8c56-4edeedce6f97-istiod-ca-cert\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156457 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-socket\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156544 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-certs\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156678 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-kube-api-access-m264v\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-kube-api-access-m264v\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156727 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/921a5139-633b-492a-8c56-4edeedce6f97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a5139-633b-492a-8c56-4edeedce6f97-istio-podinfo\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156846 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-data\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.156926 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-credential-socket\") pod \"921a5139-633b-492a-8c56-4edeedce6f97\" (UID: \"921a5139-633b-492a-8c56-4edeedce6f97\") " Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.157719 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.157745 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqbw8\" (UniqueName: \"kubernetes.io/projected/49c85304-e51f-434e-a787-bb4315eabe39-kube-api-access-mqbw8\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.157759 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c85304-e51f-434e-a787-bb4315eabe39-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.158035 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.157774 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c85304-e51f-434e-a787-bb4315eabe39-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.165158 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.162739 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-socket" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.167033 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.166836 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-credential-socket" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.167650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.167251 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-certs" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.170737 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.169764 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-istio-token" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.170737 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.170258 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921a5139-633b-492a-8c56-4edeedce6f97-istiod-ca-cert" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:49.171010 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.170965 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-data" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.172568 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.172528 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/921a5139-633b-492a-8c56-4edeedce6f97-istio-podinfo" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:49.174812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.174776 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-kube-api-access-m264v" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "kube-api-access-m264v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.176633 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.176607 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-envoy" pod "921a5139-633b-492a-8c56-4edeedce6f97" (UID: "921a5139-633b-492a-8c56-4edeedce6f97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.258999 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.258958 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.258999 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.258992 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m264v\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-kube-api-access-m264v\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.258999 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.259007 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a5139-633b-492a-8c56-4edeedce6f97-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.259493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.259019 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.259493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.259033 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.259493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.259044 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921a5139-633b-492a-8c56-4edeedce6f97-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.259493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.259056 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.259493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.259068 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a5139-633b-492a-8c56-4edeedce6f97-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.259493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.259097 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a5139-633b-492a-8c56-4edeedce6f97-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.368482 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.368450 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:57:49.415668 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.415646 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:57:49.419253 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.419232 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:57:49.462557 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462528 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-credential-socket\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.462756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462567 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-token\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-token\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.462756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462629 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istiod-ca-cert\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.462756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462677 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-envoy\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.462756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462696 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-podinfo\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.462756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462721 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-envoy\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.462756 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462756 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-data\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462827 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-kube-api-access-42xwb\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-kube-api-access-42xwb\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462845 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-credential-socket" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462878 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-token\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-token\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462926 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18017f3b-f320-4349-b79b-3e1b81bfe333-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18017f3b-f320-4349-b79b-3e1b81bfe333-istiod-ca-cert\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462946 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-podinfo\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.462991 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-credential-socket\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463007 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-socket\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463033 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb4850c-0729-48d4-95fd-758d5dfdddd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb4850c-0729-48d4-95fd-758d5dfdddd3-istiod-ca-cert\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463057 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-certs\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.463142 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463140 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-data" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463140 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-socket\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463202 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-envoy\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463299 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-certs\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463328 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-credential-socket" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463328 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-socket" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463420 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istiod-ca-cert" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463330 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-data\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463545 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-certs\") pod \"18017f3b-f320-4349-b79b-3e1b81bfe333\" (UID: \"18017f3b-f320-4349-b79b-3e1b81bfe333\") " Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463549 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-data" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463615 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-socket\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.463667 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463658 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-credential-socket\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.464220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463696 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18017f3b-f320-4349-b79b-3e1b81bfe333-istiod-ca-cert" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:49.464220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463711 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-token\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-token\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.464220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463778 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-kube-api-access-87lg7\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-kube-api-access-87lg7\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.464220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463807 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-data\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.464220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463847 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-kube-api-access-6dbtk\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-kube-api-access-6dbtk\") pod \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\" (UID: \"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23\") " Nov 01 20:57:49.464220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463915 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-podinfo\") pod \"afb4850c-0729-48d4-95fd-758d5dfdddd3\" (UID: \"afb4850c-0729-48d4-95fd-758d5dfdddd3\") " Nov 01 20:57:49.464220 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.463943 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb4850c-0729-48d4-95fd-758d5dfdddd3-istiod-ca-cert" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:49.464528 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.464309 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-certs" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465190 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465218 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb4850c-0729-48d4-95fd-758d5dfdddd3-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465233 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465250 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465265 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465280 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465298 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465312 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465326 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18017f3b-f320-4349-b79b-3e1b81bfe333-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465559 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-socket" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.465781 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-credential-socket" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.466458 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-certs" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.466530 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-podinfo" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.466639 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-certs" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.466674 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-data" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.466812 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.466783 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-socket" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.467828 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.467784 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-kube-api-access-42xwb" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "kube-api-access-42xwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.468102 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.467898 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-envoy" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.468369 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.468341 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-podinfo" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:49.469530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.469499 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-podinfo" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:49.469644 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.469579 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-token" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.469808 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.469782 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-kube-api-access-6dbtk" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "kube-api-access-6dbtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.470052 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.470035 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-token" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.470427 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.470408 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-kube-api-access-87lg7" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "kube-api-access-87lg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.470533 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.470451 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-token" pod "18017f3b-f320-4349-b79b-3e1b81bfe333" (UID: "18017f3b-f320-4349-b79b-3e1b81bfe333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.470588 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.470532 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-envoy" pod "afb4850c-0729-48d4-95fd-758d5dfdddd3" (UID: "afb4850c-0729-48d4-95fd-758d5dfdddd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.471804 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.471787 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-envoy" pod "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" (UID: "c48e99fc-dcb4-41b8-9ae7-95422d0f9a23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.567458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567415 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567449 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567466 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567484 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567500 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87lg7\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-kube-api-access-87lg7\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567516 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567529 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dbtk\" (UniqueName: \"kubernetes.io/projected/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-kube-api-access-6dbtk\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567538 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567547 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567555 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567563 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567572 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567580 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42xwb\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-kube-api-access-42xwb\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567588 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567596 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18017f3b-f320-4349-b79b-3e1b81bfe333-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567605 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567613 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.567664 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.567621 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb4850c-0729-48d4-95fd-758d5dfdddd3-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.682827 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.682784 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" event={"ID":"49c85304-e51f-434e-a787-bb4315eabe39","Type":"ContainerDied","Data":"5faabdb7d9dec6b8af9f5bc6d276e98176c3ac6344202c42489adf6da16644b0"} Nov 01 20:57:49.683786 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.682900 4477 scope.go:122] "RemoveContainer" containerID="2661f25d453e0c232205a887303de3008a15aec20511a0fb45996db6bb02fcbb" Nov 01 20:57:49.683786 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.683055 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k" Nov 01 20:57:49.686709 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.686689 4477 generic.go:356] "Generic (PLEG): container finished" podID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerID="3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b" exitCode=0 Nov 01 20:57:49.686799 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.686790 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" event={"ID":"18017f3b-f320-4349-b79b-3e1b81bfe333","Type":"ContainerDied","Data":"3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b"} Nov 01 20:57:49.686864 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.686829 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" event={"ID":"18017f3b-f320-4349-b79b-3e1b81bfe333","Type":"ContainerDied","Data":"34c42dcae1541a2127b8d08121dd328e316ec9b4ac57e48163a52ba4daf88891"} Nov 01 20:57:49.686925 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.686902 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b" Nov 01 20:57:49.714371 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.714334 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" event={"ID":"921a5139-633b-492a-8c56-4edeedce6f97","Type":"ContainerDied","Data":"bd13bcdaa67bd940e219c47cf88507e120f2eed8c93294844d80035f2c03f9eb"} Nov 01 20:57:49.714575 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.714473 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9" Nov 01 20:57:49.731137 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.729188 4477 generic.go:356] "Generic (PLEG): container finished" podID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerID="c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873" exitCode=0 Nov 01 20:57:49.731137 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.729300 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" Nov 01 20:57:49.731137 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.729313 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" event={"ID":"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23","Type":"ContainerDied","Data":"c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873"} Nov 01 20:57:49.731137 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.729366 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth" event={"ID":"c48e99fc-dcb4-41b8-9ae7-95422d0f9a23","Type":"ContainerDied","Data":"e249afbb24b4e98f9ca14d0159bc79813752a28d6f87fd0591ffd80c94e8eb23"} Nov 01 20:57:49.731675 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.731636 4477 generic.go:356] "Generic (PLEG): container finished" podID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerID="0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8" exitCode=0 Nov 01 20:57:49.731803 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.731686 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" Nov 01 20:57:49.731803 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.731763 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" event={"ID":"afb4850c-0729-48d4-95fd-758d5dfdddd3","Type":"ContainerDied","Data":"0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8"} Nov 01 20:57:49.731911 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.731805 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c" event={"ID":"afb4850c-0729-48d4-95fd-758d5dfdddd3","Type":"ContainerDied","Data":"27f0bc2167806d0e30c354b1572062bfa5d5b2c3b1e1fb13e628e7b11635c543"} Nov 01 20:57:49.734785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.734762 4477 scope.go:122] "RemoveContainer" containerID="3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b" Nov 01 20:57:49.740194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.739698 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" Nov 01 20:57:49.740194 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.739730 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt" event={"ID":"f3cc78a3-f034-4873-9e85-0481a94d6818","Type":"ContainerDied","Data":"437b22b35bf8a202e96e5625a219a599018c060c52abc916265bffd4aa293bea"} Nov 01 20:57:49.755205 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.755182 4477 scope.go:122] "RemoveContainer" containerID="3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b" Nov 01 20:57:49.755560 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:57:49.755542 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b\": container with ID starting with 3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b not found: ID does not exist" containerID="3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b" Nov 01 20:57:49.755647 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.755572 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b"} err="failed to get container status \"3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b\": rpc error: code = NotFound desc = could not find container \"3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b\": container with ID starting with 3993a78e569c08b959a71447fa27ce0a4b4131b0bcf1424c6a36bbb428dc378b not found: ID does not exist" Nov 01 20:57:49.755647 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.755596 4477 scope.go:122] "RemoveContainer" containerID="3122c958df5be523e7f158a94de52086f2e3c274bbbebef2d3f00c501a385dbd" Nov 01 20:57:49.768020 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.768003 4477 scope.go:122] "RemoveContainer" containerID="c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873" Nov 01 20:57:49.777744 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.777726 4477 scope.go:122] "RemoveContainer" containerID="c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873" Nov 01 20:57:49.777991 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:57:49.777972 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873\": container with ID starting with c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873 not found: ID does not exist" containerID="c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873" Nov 01 20:57:49.778057 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.777997 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873"} err="failed to get container status \"c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873\": rpc error: code = NotFound desc = could not find container \"c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873\": container with ID starting with c5215a0a057308b795eab1fcf4b9ec155f83c7d306245d1d60e630ff4161c873 not found: ID does not exist" Nov 01 20:57:49.778057 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.778012 4477 scope.go:122] "RemoveContainer" containerID="0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8" Nov 01 20:57:49.787063 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.787048 4477 scope.go:122] "RemoveContainer" containerID="0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8" Nov 01 20:57:49.787330 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:57:49.787309 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8\": container with ID starting with 0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8 not found: ID does not exist" containerID="0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8" Nov 01 20:57:49.787419 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.787334 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8"} err="failed to get container status \"0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8\": rpc error: code = NotFound desc = could not find container \"0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8\": container with ID starting with 0c108779f7238f315c308084cff609d2a108a54cbdb3f2254326a154dd373cb8 not found: ID does not exist" Nov 01 20:57:49.787419 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.787350 4477 scope.go:122] "RemoveContainer" containerID="ce01ca70074e110490301b0840eaeb5ee10c170fcb4780ffbd37f620316b5519" Nov 01 20:57:49.844947 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.844867 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b"] Nov 01 20:57:49.851311 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:49.851268 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-7rr6b"] Nov 01 20:57:50.005236 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.005204 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k"] Nov 01 20:57:50.009701 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.009678 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-mfk4k"] Nov 01 20:57:50.117702 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.117608 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth"] Nov 01 20:57:50.121765 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.121737 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-f7qth"] Nov 01 20:57:50.229776 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.229743 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt"] Nov 01 20:57:50.234230 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.234190 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-gx9rt"] Nov 01 20:57:50.341357 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.341311 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9"] Nov 01 20:57:50.345377 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.345354 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-jlkc9"] Nov 01 20:57:50.453124 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.453014 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c"] Nov 01 20:57:50.456932 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.456912 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8wf8c"] Nov 01 20:57:50.505773 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.505742 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" path="/var/lib/kubelet/pods/18017f3b-f320-4349-b79b-3e1b81bfe333/volumes" Nov 01 20:57:50.506292 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.506278 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c85304-e51f-434e-a787-bb4315eabe39" path="/var/lib/kubelet/pods/49c85304-e51f-434e-a787-bb4315eabe39/volumes" Nov 01 20:57:50.506754 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.506741 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921a5139-633b-492a-8c56-4edeedce6f97" path="/var/lib/kubelet/pods/921a5139-633b-492a-8c56-4edeedce6f97/volumes" Nov 01 20:57:50.507529 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.507515 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" path="/var/lib/kubelet/pods/afb4850c-0729-48d4-95fd-758d5dfdddd3/volumes" Nov 01 20:57:50.507944 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.507932 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" path="/var/lib/kubelet/pods/c48e99fc-dcb4-41b8-9ae7-95422d0f9a23/volumes" Nov 01 20:57:50.508402 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:50.508388 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" path="/var/lib/kubelet/pods/f3cc78a3-f034-4873-9e85-0481a94d6818/volumes" Nov 01 20:57:59.355197 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:59.355154 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv"] Nov 01 20:57:59.355675 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:57:59.355470 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" containerID="cri-o://eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2" gracePeriod=30 Nov 01 20:58:02.337810 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:02.337749 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:58:04.028100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.027649 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:58:04.028100 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.027731 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:58:04.815405 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.815373 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:58:04.934680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.934599 4477 generic.go:356] "Generic (PLEG): container finished" podID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerID="eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2" exitCode=0 Nov 01 20:58:04.934680 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.934644 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" Nov 01 20:58:04.934865 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.934686 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" event={"ID":"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13","Type":"ContainerDied","Data":"eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2"} Nov 01 20:58:04.934865 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.934741 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv" event={"ID":"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13","Type":"ContainerDied","Data":"5c5f84c4d2747209bb83b7ec0d262a5602e7ffeb8b3f80ecdeba29c3b32cd1d4"} Nov 01 20:58:04.934865 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.934769 4477 scope.go:122] "RemoveContainer" containerID="eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2" Nov 01 20:58:04.945530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.945511 4477 scope.go:122] "RemoveContainer" containerID="eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2" Nov 01 20:58:04.945810 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:04.945791 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2\": container with ID starting with eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2 not found: ID does not exist" containerID="eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2" Nov 01 20:58:04.945857 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.945819 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2"} err="failed to get container status \"eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2\": rpc error: code = NotFound desc = could not find container \"eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2\": container with ID starting with eb4ec68c4d557c95d96f3ef60138d9bdd30045e86cd33e81cc73f6eb94d7cde2 not found: ID does not exist" Nov 01 20:58:04.958530 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958506 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-credential-socket\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.958646 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958572 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-envoy\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.958688 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958661 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istiod-ca-cert\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.958746 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958701 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-credential-socket" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.958746 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958735 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-data\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.958837 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958764 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-certs\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.958930 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958833 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-podinfo\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.958930 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958898 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-token\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-token\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.959187 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958950 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-kube-api-access-gshs9\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-kube-api-access-gshs9\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.959187 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958990 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-socket\") pod \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\" (UID: \"bba27e49-0af7-4f7e-8e2e-1ecf2190cc13\") " Nov 01 20:58:04.959187 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.958993 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-data" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.959187 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.959051 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istiod-ca-cert" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:04.959456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.959409 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-certs" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.959456 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.959445 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-socket" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.959946 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.959924 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.960053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.959952 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.960053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.959968 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.960053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.959983 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.960053 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.959996 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.961201 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.961176 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-envoy" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.961398 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.961369 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-podinfo" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:04.961493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.961454 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-token" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:04.961493 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:04.961484 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-kube-api-access-gshs9" pod "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" (UID: "bba27e49-0af7-4f7e-8e2e-1ecf2190cc13"). InnerVolumeSpecName "kube-api-access-gshs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:05.061903 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:05.061869 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.061903 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:05.061900 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.061903 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:05.061910 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gshs9\" (UniqueName: \"kubernetes.io/projected/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-kube-api-access-gshs9\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.062347 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:05.061920 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.361256 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:05.361225 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv"] Nov 01 20:58:05.365590 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:05.365569 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vnzfv"] Nov 01 20:58:06.505239 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:06.505207 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" path="/var/lib/kubelet/pods/bba27e49-0af7-4f7e-8e2e-1ecf2190cc13/volumes" Nov 01 20:58:06.554683 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:06.554648 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:58:07.020183 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:07.020150 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v"] Nov 01 20:58:07.020387 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:07.020366 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" containerID="cri-o://5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee" gracePeriod=30 Nov 01 20:58:07.034066 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:07.034036 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx"] Nov 01 20:58:07.036598 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:07.036031 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" containerID="cri-o://4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e" gracePeriod=30 Nov 01 20:58:07.964413 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:07.964374 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:58:07.964792 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:07.964451 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:58:08.665984 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:08.665939 4477 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:58:08.666237 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:08.666002 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:58:12.363200 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.363138 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:58:12.405711 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.405684 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:58:12.471191 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471160 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-kube-api-access-rbcvb\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-kube-api-access-rbcvb\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.471441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471292 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-credential-socket\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.471441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471317 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-data\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.471441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471351 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istiod-ca-cert\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.471441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471404 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-podinfo\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.471661 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471538 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-token\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-token\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.471661 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471602 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-envoy\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.471785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471648 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-data" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.471785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471664 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-credential-socket" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.471785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471703 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-certs\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.471785 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.471744 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-socket\") pod \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\" (UID: \"0149e00e-0b29-4be7-bd28-0dbdf08a338a\") " Nov 01 20:58:12.472112 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.472062 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-socket" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.472659 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.472611 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.472659 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.472625 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istiod-ca-cert" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:12.472659 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.472643 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.472659 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.472658 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.473298 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.473120 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-certs" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.474382 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.474340 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-kube-api-access-rbcvb" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "kube-api-access-rbcvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:12.474586 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.474552 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-envoy" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.474822 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.474803 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-token" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:12.474894 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.474864 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-podinfo" pod "0149e00e-0b29-4be7-bd28-0dbdf08a338a" (UID: "0149e00e-0b29-4be7-bd28-0dbdf08a338a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:12.574183 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574050 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-socket\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574206 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-kube-api-access-dbk2l\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-kube-api-access-dbk2l\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574289 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-socket" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.574365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574310 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-podinfo\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574365 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574344 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istiod-ca-cert\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574465 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-token\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-token\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574535 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-data\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574581 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-certs\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574754 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574608 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-envoy\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574754 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574701 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-credential-socket\") pod \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\" (UID: \"a3a3a587-3e33-4355-b2e4-ae18cd8291da\") " Nov 01 20:58:12.574754 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574727 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istiod-ca-cert" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:12.574899 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574824 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-data" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.575089 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.574997 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-certs" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.575416 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575393 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-credential-socket" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.575727 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575708 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbcvb\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-kube-api-access-rbcvb\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.575798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575734 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.575798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575750 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.575798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575764 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.575798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575777 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.575798 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575794 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.576057 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575809 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.576057 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575823 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.576057 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575838 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.576057 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575851 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149e00e-0b29-4be7-bd28-0dbdf08a338a-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.576057 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.575864 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.576779 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.576750 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-podinfo" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:12.576779 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.576764 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-token" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:12.576907 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.576810 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-kube-api-access-dbk2l" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "kube-api-access-dbk2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:12.577875 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.577857 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-envoy" pod "a3a3a587-3e33-4355-b2e4-ae18cd8291da" (UID: "a3a3a587-3e33-4355-b2e4-ae18cd8291da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.677631 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.677595 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.677631 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.677630 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbk2l\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-kube-api-access-dbk2l\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.677846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.677643 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.677846 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:12.677651 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a3a587-3e33-4355-b2e4-ae18cd8291da-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:13.015409 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.015323 4477 generic.go:356] "Generic (PLEG): container finished" podID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerID="4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e" exitCode=0 Nov 01 20:58:13.015409 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.015368 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" Nov 01 20:58:13.015608 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.015413 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" event={"ID":"a3a3a587-3e33-4355-b2e4-ae18cd8291da","Type":"ContainerDied","Data":"4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e"} Nov 01 20:58:13.015608 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.015479 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx" event={"ID":"a3a3a587-3e33-4355-b2e4-ae18cd8291da","Type":"ContainerDied","Data":"4ed0cea2329e6e39d42e9708fcc1a824a61635d9438331b2127040f928ed5eca"} Nov 01 20:58:13.015608 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.015512 4477 scope.go:122] "RemoveContainer" containerID="4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e" Nov 01 20:58:13.017488 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.017461 4477 generic.go:356] "Generic (PLEG): container finished" podID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerID="5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee" exitCode=0 Nov 01 20:58:13.017612 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.017517 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" Nov 01 20:58:13.017612 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.017555 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" event={"ID":"0149e00e-0b29-4be7-bd28-0dbdf08a338a","Type":"ContainerDied","Data":"5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee"} Nov 01 20:58:13.017722 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.017616 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v" event={"ID":"0149e00e-0b29-4be7-bd28-0dbdf08a338a","Type":"ContainerDied","Data":"9e0c411c7ec8a6fb304fa9f9f1075597cf61a5dcbd3767a0ab1631cf4aced22c"} Nov 01 20:58:13.028063 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.028048 4477 scope.go:122] "RemoveContainer" containerID="4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e" Nov 01 20:58:13.029027 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:13.028994 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e\": container with ID starting with 4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e not found: ID does not exist" containerID="4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e" Nov 01 20:58:13.029200 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.029176 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e"} err="failed to get container status \"4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e\": rpc error: code = NotFound desc = could not find container \"4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e\": container with ID starting with 4f7c1076b27387624c1b9b35098b027e450c2bd23c6c12be036aa9c142880a3e not found: ID does not exist" Nov 01 20:58:13.029315 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.029305 4477 scope.go:122] "RemoveContainer" containerID="5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee" Nov 01 20:58:13.043152 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.043128 4477 scope.go:122] "RemoveContainer" containerID="5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee" Nov 01 20:58:13.043485 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:13.043463 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee\": container with ID starting with 5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee not found: ID does not exist" containerID="5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee" Nov 01 20:58:13.043572 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.043498 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee"} err="failed to get container status \"5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee\": rpc error: code = NotFound desc = could not find container \"5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee\": container with ID starting with 5331137a84acc6b7b1c41a58ab05974798c2004d8091af41fbf41402cee74dee not found: ID does not exist" Nov 01 20:58:13.182816 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.182782 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx"] Nov 01 20:58:13.189265 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.189235 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-g2rzx"] Nov 01 20:58:13.291954 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.291857 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v"] Nov 01 20:58:13.295524 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:13.295500 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6b97v"] Nov 01 20:58:14.505463 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:14.505432 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" path="/var/lib/kubelet/pods/0149e00e-0b29-4be7-bd28-0dbdf08a338a/volumes" Nov 01 20:58:14.506154 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:14.506135 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" path="/var/lib/kubelet/pods/a3a3a587-3e33-4355-b2e4-ae18cd8291da/volumes" Nov 01 20:58:36.555236 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:36.555145 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:58:37.287595 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.287563 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm"] Nov 01 20:58:37.288183 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288170 4477 cpu_manager.go:414] "containerMap: removing container" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" Nov 01 20:58:37.288235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288185 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" Nov 01 20:58:37.288235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288206 4477 cpu_manager.go:414] "containerMap: removing container" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" Nov 01 20:58:37.288235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288211 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" Nov 01 20:58:37.288235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288218 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" Nov 01 20:58:37.288235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288223 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" Nov 01 20:58:37.288235 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288234 4477 cpu_manager.go:414] "containerMap: removing container" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288238 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288249 4477 cpu_manager.go:414] "containerMap: removing container" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288253 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288260 4477 cpu_manager.go:414] "containerMap: removing container" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288264 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288270 4477 cpu_manager.go:414] "containerMap: removing container" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288274 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288281 4477 cpu_manager.go:414] "containerMap: removing container" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288286 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288296 4477 cpu_manager.go:414] "containerMap: removing container" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288300 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288388 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0149e00e-0b29-4be7-bd28-0dbdf08a338a" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288399 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="921a5139-633b-492a-8c56-4edeedce6f97" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288406 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a3a587-3e33-4355-b2e4-ae18cd8291da" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288413 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="18017f3b-f320-4349-b79b-3e1b81bfe333" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288420 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c85304-e51f-434e-a787-bb4315eabe39" containerName="istio-proxy" Nov 01 20:58:37.288420 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288425 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48e99fc-dcb4-41b8-9ae7-95422d0f9a23" containerName="istio-proxy" Nov 01 20:58:37.288901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288431 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cc78a3-f034-4873-9e85-0481a94d6818" containerName="istio-proxy" Nov 01 20:58:37.288901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288437 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba27e49-0af7-4f7e-8e2e-1ecf2190cc13" containerName="istio-proxy" Nov 01 20:58:37.288901 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.288442 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb4850c-0729-48d4-95fd-758d5dfdddd3" containerName="istio-proxy" Nov 01 20:58:37.291994 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.291978 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.357324 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.357289 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm"] Nov 01 20:58:37.382795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.382760 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlc2\" (UniqueName: \"kubernetes.io/projected/760a62bd-4672-4367-a6df-55b3a23b13c2-kube-api-access-tqlc2\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.382970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.382865 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.382970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.382891 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760a62bd-4672-4367-a6df-55b3a23b13c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.382970 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.382914 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.392850 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.392822 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-de8145-serving-cert\"" Nov 01 20:58:37.484363 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.484327 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.484580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.484391 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760a62bd-4672-4367-a6df-55b3a23b13c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.484580 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.484424 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.484580 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:37.484461 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-serving-cert: secret "maas-api-e2e-shared-a-w0-de8145-serving-cert" not found Nov 01 20:58:37.484580 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:37.484533 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls podName:760a62bd-4672-4367-a6df-55b3a23b13c2 nodeName:}" failed. No retries permitted until 2026-11-01 20:58:37.984511359 +0000 UTC m=+1671.634562887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls") pod "maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" (UID: "760a62bd-4672-4367-a6df-55b3a23b13c2") : secret "maas-api-e2e-shared-a-w0-de8145-serving-cert" not found Nov 01 20:58:37.484835 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.484814 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlc2\" (UniqueName: \"kubernetes.io/projected/760a62bd-4672-4367-a6df-55b3a23b13c2-kube-api-access-tqlc2\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.485143 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.485118 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760a62bd-4672-4367-a6df-55b3a23b13c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.487523 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.487495 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.549243 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.549160 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlc2\" (UniqueName: \"kubernetes.io/projected/760a62bd-4672-4367-a6df-55b3a23b13c2-kube-api-access-tqlc2\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.991956 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.991917 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:37.994554 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:37.994531 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:38.206628 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:38.206592 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:38.591345 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:38.591316 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm"] Nov 01 20:58:38.591861 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:58:38.591820 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760a62bd_4672_4367_a6df_55b3a23b13c2.slice/crio-826c60bdec9f853f77fe6b77b550f7df3347dafaf6c8196cba0afe95782fc098 WatchSource:0}: Error finding container 826c60bdec9f853f77fe6b77b550f7df3347dafaf6c8196cba0afe95782fc098: Status 404 returned error can't find the container with id 826c60bdec9f853f77fe6b77b550f7df3347dafaf6c8196cba0afe95782fc098 Nov 01 20:58:39.362412 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:39.362371 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" event={"ID":"760a62bd-4672-4367-a6df-55b3a23b13c2","Type":"ContainerStarted","Data":"826c60bdec9f853f77fe6b77b550f7df3347dafaf6c8196cba0afe95782fc098"} Nov 01 20:58:40.372917 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:40.372867 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" event={"ID":"760a62bd-4672-4367-a6df-55b3a23b13c2","Type":"ContainerStarted","Data":"9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344"} Nov 01 20:58:40.377160 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:40.374176 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:40.492094 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:40.491977 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" podStartSLOduration=2.562581851 podStartE2EDuration="3.491961957s" totalImagesPullingTime="929.380106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:40.488942984 +0000 UTC m=+1674.138994569" watchObservedRunningTime="2026-11-01 20:58:40.491961957 +0000 UTC m=+1674.142013536" Nov 01 20:58:47.396306 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:47.396276 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 20:58:47.972959 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:47.972929 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8"] Nov 01 20:58:47.978961 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:47.978939 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.031091 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.031043 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-a9b439-serving-cert\"" Nov 01 20:58:48.035641 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.035608 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8"] Nov 01 20:58:48.084462 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.084427 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.085137 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.085052 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.085474 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.085454 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84j6c\" (UniqueName: \"kubernetes.io/projected/1370f293-b39b-49da-97e8-92458dcf6e52-kube-api-access-84j6c\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.085712 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.085694 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370f293-b39b-49da-97e8-92458dcf6e52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.209104 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.187353 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.219192 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.219139 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84j6c\" (UniqueName: \"kubernetes.io/projected/1370f293-b39b-49da-97e8-92458dcf6e52-kube-api-access-84j6c\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.219578 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.219519 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370f293-b39b-49da-97e8-92458dcf6e52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.223952 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.220238 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.224354 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:48.197095 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-serving-cert: secret "maas-api-e2e-shared-a-w4-a9b439-serving-cert" not found Nov 01 20:58:48.224518 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:58:48.224409 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls podName:1370f293-b39b-49da-97e8-92458dcf6e52 nodeName:}" failed. No retries permitted until 2026-11-01 20:58:48.724385809 +0000 UTC m=+1682.374437359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls") pod "maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" (UID: "1370f293-b39b-49da-97e8-92458dcf6e52") : secret "maas-api-e2e-shared-a-w4-a9b439-serving-cert" not found Nov 01 20:58:48.225214 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.225171 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370f293-b39b-49da-97e8-92458dcf6e52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.233131 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.228812 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.356778 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.356741 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84j6c\" (UniqueName: \"kubernetes.io/projected/1370f293-b39b-49da-97e8-92458dcf6e52-kube-api-access-84j6c\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.758748 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.758707 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.761374 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.761348 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:48.894395 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:48.894355 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:49.284312 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:49.284287 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8"] Nov 01 20:58:49.284864 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:58:49.284844 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1370f293_b39b_49da_97e8_92458dcf6e52.slice/crio-ef7e1efdc5e77d644ca20680037e7829aadf4b5cd59d7c22782cd58f41e0b43b WatchSource:0}: Error finding container ef7e1efdc5e77d644ca20680037e7829aadf4b5cd59d7c22782cd58f41e0b43b: Status 404 returned error can't find the container with id ef7e1efdc5e77d644ca20680037e7829aadf4b5cd59d7c22782cd58f41e0b43b Nov 01 20:58:49.460270 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:49.460239 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" event={"ID":"1370f293-b39b-49da-97e8-92458dcf6e52","Type":"ContainerStarted","Data":"ef7e1efdc5e77d644ca20680037e7829aadf4b5cd59d7c22782cd58f41e0b43b"} Nov 01 20:58:50.491925 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:50.488658 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" event={"ID":"1370f293-b39b-49da-97e8-92458dcf6e52","Type":"ContainerStarted","Data":"ea5577a98ec0cf56c5317ee98c33807a3e0504e918397a0a056e4502b7621730"} Nov 01 20:58:50.491925 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:50.489786 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:58:50.607581 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:50.607527 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" podStartSLOduration=2.726361477 podStartE2EDuration="3.607512548s" totalImagesPullingTime="881.151071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:50.604155937 +0000 UTC m=+1684.254207502" watchObservedRunningTime="2026-11-01 20:58:50.607512548 +0000 UTC m=+1684.257564116" Nov 01 20:58:57.516453 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:58:57.516425 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 20:59:06.555894 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:59:06.555857 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:59:07.547120 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:07.546087 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:59:21.560087 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.560030 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz"] Nov 01 20:59:21.564811 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.564786 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.626149 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.620353 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz"] Nov 01 20:59:21.665478 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.665447 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b27e66-serving-cert\"" Nov 01 20:59:21.747173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.747139 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.747173 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.747172 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstwt\" (UniqueName: \"kubernetes.io/projected/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-kube-api-access-pstwt\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.747421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.747264 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-metrics-certs\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.747421 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.747335 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-openshift-service-ca\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.850790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.849979 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.850790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.850028 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstwt\" (UniqueName: \"kubernetes.io/projected/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-kube-api-access-pstwt\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.850790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.850123 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-metrics-certs\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.850790 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.850205 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-openshift-service-ca\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.851735 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.851711 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-openshift-service-ca\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.851973 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:59:21.851957 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-serving-cert: secret "maas-api-e2e-api-a-b27e66-serving-cert" not found Nov 01 20:59:21.852140 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:59:21.852129 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls podName:c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016 nodeName:}" failed. No retries permitted until 2026-11-01 20:59:22.352109007 +0000 UTC m=+1716.002160539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls") pod "maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" (UID: "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016") : secret "maas-api-e2e-api-a-b27e66-serving-cert" not found Nov 01 20:59:21.857506 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.857481 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-metrics-certs\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:21.915068 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:21.915037 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstwt\" (UniqueName: \"kubernetes.io/projected/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-kube-api-access-pstwt\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:22.357945 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:22.357906 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:22.360795 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:22.360762 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls\") pod \"maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:22.478548 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:22.478515 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:22.869749 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:22.869721 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz"] Nov 01 20:59:22.870143 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:59:22.869985 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fca6f1_9a04_4a5d_bb8d_718ebe2d8016.slice/crio-49bf7353fe1bc47b429f1f73a38999afa9e81737139796026ff52d7bb9b16a2b WatchSource:0}: Error finding container 49bf7353fe1bc47b429f1f73a38999afa9e81737139796026ff52d7bb9b16a2b: Status 404 returned error can't find the container with id 49bf7353fe1bc47b429f1f73a38999afa9e81737139796026ff52d7bb9b16a2b Nov 01 20:59:23.753159 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:23.753120 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" event={"ID":"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016","Type":"ContainerStarted","Data":"49bf7353fe1bc47b429f1f73a38999afa9e81737139796026ff52d7bb9b16a2b"} Nov 01 20:59:24.762643 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:24.762600 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" event={"ID":"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016","Type":"ContainerStarted","Data":"ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736"} Nov 01 20:59:24.763206 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:24.762774 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:24.891028 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:24.890968 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" podStartSLOduration=2.97272747 podStartE2EDuration="3.890951997s" totalImagesPullingTime="918.224527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:59:24.888554477 +0000 UTC m=+1718.538606043" watchObservedRunningTime="2026-11-01 20:59:24.890951997 +0000 UTC m=+1718.541003541" Nov 01 20:59:26.404106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.404056 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8"] Nov 01 20:59:26.408488 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.408468 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.465800 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.465766 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8"] Nov 01 20:59:26.509464 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.509427 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-df700a-serving-cert\"" Nov 01 20:59:26.518336 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.518307 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6ef13d-442d-48e5-9a43-905aa13428a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.518336 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.518340 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmfs\" (UniqueName: \"kubernetes.io/projected/ad6ef13d-442d-48e5-9a43-905aa13428a3-kube-api-access-xvmfs\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.518600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.518510 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.518600 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.518552 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.619808 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.619772 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6ef13d-442d-48e5-9a43-905aa13428a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.619808 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.619810 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmfs\" (UniqueName: \"kubernetes.io/projected/ad6ef13d-442d-48e5-9a43-905aa13428a3-kube-api-access-xvmfs\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.620106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.619948 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.620106 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.620009 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.620223 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:59:26.620104 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-serving-cert: secret "maas-api-e2e-shared-b-w1-df700a-serving-cert" not found Nov 01 20:59:26.620223 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:59:26.620182 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls podName:ad6ef13d-442d-48e5-9a43-905aa13428a3 nodeName:}" failed. No retries permitted until 2026-11-01 20:59:27.12016034 +0000 UTC m=+1720.770211878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls") pod "maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" (UID: "ad6ef13d-442d-48e5-9a43-905aa13428a3") : secret "maas-api-e2e-shared-b-w1-df700a-serving-cert" not found Nov 01 20:59:26.620521 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.620502 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6ef13d-442d-48e5-9a43-905aa13428a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.622766 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.622743 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:26.679542 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:26.679449 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmfs\" (UniqueName: \"kubernetes.io/projected/ad6ef13d-442d-48e5-9a43-905aa13428a3-kube-api-access-xvmfs\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:27.127658 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:27.127627 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:27.130295 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:27.130265 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:27.322689 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:27.322658 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:27.708452 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:27.708421 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8"] Nov 01 20:59:27.787858 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:27.787821 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" event={"ID":"ad6ef13d-442d-48e5-9a43-905aa13428a3","Type":"ContainerStarted","Data":"a199c5e6c204ebe5a672d0c729c884d7d1bc992dc47135086508738ac05c9d0f"} Nov 01 20:59:28.798591 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:28.798557 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" event={"ID":"ad6ef13d-442d-48e5-9a43-905aa13428a3","Type":"ContainerStarted","Data":"05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255"} Nov 01 20:59:28.799107 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:28.798634 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:28.920773 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:28.920629 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" podStartSLOduration=2.02742941 podStartE2EDuration="2.920610518s" totalImagesPullingTime="893.181108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:59:28.913945437 +0000 UTC m=+1722.563997028" watchObservedRunningTime="2026-11-01 20:59:28.920610518 +0000 UTC m=+1722.570662064" Nov 01 20:59:30.774990 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:30.774908 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 20:59:34.812065 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:34.812031 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 20:59:36.563689 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:59:36.563648 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 20:59:49.922007 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:49.921968 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 20:59:55.361914 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.361868 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x"] Nov 01 20:59:55.368569 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.368532 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.421604 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.421567 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x"] Nov 01 20:59:55.464394 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.464359 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-bf3fa2-serving-cert\"" Nov 01 20:59:55.490269 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.490222 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw282\" (UniqueName: \"kubernetes.io/projected/23c4a40e-579b-4957-abe7-83fd4f082509-kube-api-access-cw282\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.490458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.490272 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.490458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.490374 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c4a40e-579b-4957-abe7-83fd4f082509-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.490589 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.490494 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.591586 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.591547 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.591773 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:59:55.591703 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-serving-cert: secret "maas-api-e2e-shared-b-w3-bf3fa2-serving-cert" not found Nov 01 20:59:55.591834 ip-10-0-133-242 kubenswrapper[4477]: E1101 20:59:55.591778 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls podName:23c4a40e-579b-4957-abe7-83fd4f082509 nodeName:}" failed. No retries permitted until 2026-11-01 20:59:56.091760573 +0000 UTC m=+1749.741812096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls") pod "maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" (UID: "23c4a40e-579b-4957-abe7-83fd4f082509") : secret "maas-api-e2e-shared-b-w3-bf3fa2-serving-cert" not found Nov 01 20:59:55.591922 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.591906 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw282\" (UniqueName: \"kubernetes.io/projected/23c4a40e-579b-4957-abe7-83fd4f082509-kube-api-access-cw282\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.591982 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.591939 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.591982 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.591962 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c4a40e-579b-4957-abe7-83fd4f082509-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.592659 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.592639 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c4a40e-579b-4957-abe7-83fd4f082509-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.594471 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.594448 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:55.652225 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:55.652110 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw282\" (UniqueName: \"kubernetes.io/projected/23c4a40e-579b-4957-abe7-83fd4f082509-kube-api-access-cw282\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:56.099860 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:56.099819 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:56.102576 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:56.102538 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:56.283303 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:56.283267 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:56.672751 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:56.672722 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x"] Nov 01 20:59:56.673882 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:59:56.673849 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c4a40e_579b_4957_abe7_83fd4f082509.slice/crio-73147aa5d22cfb2fd61f4523e10edd5baa9f10f6de068d9c1b95c302ba459495 WatchSource:0}: Error finding container 73147aa5d22cfb2fd61f4523e10edd5baa9f10f6de068d9c1b95c302ba459495: Status 404 returned error can't find the container with id 73147aa5d22cfb2fd61f4523e10edd5baa9f10f6de068d9c1b95c302ba459495 Nov 01 20:59:57.056650 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:57.056556 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" event={"ID":"23c4a40e-579b-4957-abe7-83fd4f082509","Type":"ContainerStarted","Data":"73147aa5d22cfb2fd61f4523e10edd5baa9f10f6de068d9c1b95c302ba459495"} Nov 01 20:59:58.065399 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.065366 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" event={"ID":"23c4a40e-579b-4957-abe7-83fd4f082509","Type":"ContainerStarted","Data":"e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2"} Nov 01 20:59:58.065772 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.065422 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 20:59:58.182294 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.182237 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" podStartSLOduration=2.222438696 podStartE2EDuration="3.182218794s" totalImagesPullingTime="959.780098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:59:58.178318746 +0000 UTC m=+1751.828370304" watchObservedRunningTime="2026-11-01 20:59:58.182218794 +0000 UTC m=+1751.832270341" Nov 01 20:59:58.478444 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.478352 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg"] Nov 01 20:59:58.478627 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.478567 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" containerID="cri-o://01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199" gracePeriod=30 Nov 01 20:59:58.530458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.529692 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k"] Nov 01 20:59:58.530458 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.529915 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" containerID="cri-o://15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7" gracePeriod=30 Nov 01 20:59:58.641635 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.641598 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7304-eb614d"] Nov 01 20:59:58.646011 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.645989 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-eb614d" Nov 01 20:59:58.706234 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.705464 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8mhs\"" Nov 01 20:59:58.706234 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.705806 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7304-eb614d"] Nov 01 20:59:58.744977 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.744270 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55twj\" (UniqueName: \"kubernetes.io/projected/b2888814-e8f2-48bb-acd9-5db0d227232b-kube-api-access-55twj\") pod \"test-curl-7304-eb614d\" (UID: \"b2888814-e8f2-48bb-acd9-5db0d227232b\") " pod="opendatahub/test-curl-7304-eb614d" Nov 01 20:59:58.814767 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.814731 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6"] Nov 01 20:59:58.815000 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.814978 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" containerID="cri-o://0688182a850706f0766858beba3d7658277e3e0161a3eecfa8029339121feaaf" gracePeriod=30 Nov 01 20:59:58.845948 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.845903 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55twj\" (UniqueName: \"kubernetes.io/projected/b2888814-e8f2-48bb-acd9-5db0d227232b-kube-api-access-55twj\") pod \"test-curl-7304-eb614d\" (UID: \"b2888814-e8f2-48bb-acd9-5db0d227232b\") " pod="opendatahub/test-curl-7304-eb614d" Nov 01 20:59:58.903740 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.903704 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55twj\" (UniqueName: \"kubernetes.io/projected/b2888814-e8f2-48bb-acd9-5db0d227232b-kube-api-access-55twj\") pod \"test-curl-7304-eb614d\" (UID: \"b2888814-e8f2-48bb-acd9-5db0d227232b\") " pod="opendatahub/test-curl-7304-eb614d" Nov 01 20:59:58.961441 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:58.961401 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-eb614d" Nov 01 20:59:59.137992 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:59.137955 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2"] Nov 01 20:59:59.138655 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:59.138280 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" containerID="cri-o://12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b" gracePeriod=30 Nov 01 20:59:59.351097 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:59.350460 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7304-eb614d"] Nov 01 20:59:59.358027 ip-10-0-133-242 kubenswrapper[4477]: W1101 20:59:59.357989 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2888814_e8f2_48bb_acd9_5db0d227232b.slice/crio-73e5364c26488f2f8423eb4c83daed7a7b098712572cf3fe1f3cb72d344b21e6 WatchSource:0}: Error finding container 73e5364c26488f2f8423eb4c83daed7a7b098712572cf3fe1f3cb72d344b21e6: Status 404 returned error can't find the container with id 73e5364c26488f2f8423eb4c83daed7a7b098712572cf3fe1f3cb72d344b21e6 Nov 01 20:59:59.435490 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:59.435416 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:59:59.435490 ip-10-0-133-242 kubenswrapper[4477]: I1101 20:59:59.435470 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:00:00.082684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.082643 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7304-eb614d" event={"ID":"b2888814-e8f2-48bb-acd9-5db0d227232b","Type":"ContainerStarted","Data":"f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc"} Nov 01 21:00:00.082869 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.082696 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7304-eb614d" event={"ID":"b2888814-e8f2-48bb-acd9-5db0d227232b","Type":"ContainerStarted","Data":"73e5364c26488f2f8423eb4c83daed7a7b098712572cf3fe1f3cb72d344b21e6"} Nov 01 21:00:00.202417 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.202354 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7304-eb614d" podStartSLOduration=2.202334376 podStartE2EDuration="2.202334376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:00.196707739 +0000 UTC m=+1753.846759283" watchObservedRunningTime="2026-11-01 21:00:00.202334376 +0000 UTC m=+1753.852385921" Nov 01 21:00:00.219131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.219096 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5"] Nov 01 21:00:00.227148 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.227119 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:00.283874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.283838 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5"] Nov 01 21:00:00.347170 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.347046 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:00:00.370368 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.370329 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa181ab-d548-4a6e-8aa4-ffe4189484a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5\" (UID: \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:00.370651 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.370429 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvjg\" (UniqueName: \"kubernetes.io/projected/faa181ab-d548-4a6e-8aa4-ffe4189484a3-kube-api-access-tfvjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5\" (UID: \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:00.472216 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.472175 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa181ab-d548-4a6e-8aa4-ffe4189484a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5\" (UID: \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:00.472369 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.472322 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvjg\" (UniqueName: \"kubernetes.io/projected/faa181ab-d548-4a6e-8aa4-ffe4189484a3-kube-api-access-tfvjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5\" (UID: \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:00.472857 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.472840 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa181ab-d548-4a6e-8aa4-ffe4189484a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5\" (UID: \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:00.518674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.518449 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:00:00.518674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.518522 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:00:00.530138 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.530105 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvjg\" (UniqueName: \"kubernetes.io/projected/faa181ab-d548-4a6e-8aa4-ffe4189484a3-kube-api-access-tfvjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5\" (UID: \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:00.548438 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.548395 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:00.931828 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:00.931791 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5"] Nov 01 21:00:00.931966 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:00:00.931851 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa181ab_d548_4a6e_8aa4_ffe4189484a3.slice/crio-1a10a5544320a4aa992598068cfcb27213bb7cf946ac69750b67ddce901859de WatchSource:0}: Error finding container 1a10a5544320a4aa992598068cfcb27213bb7cf946ac69750b67ddce901859de: Status 404 returned error can't find the container with id 1a10a5544320a4aa992598068cfcb27213bb7cf946ac69750b67ddce901859de Nov 01 21:00:01.093459 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:01.093415 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" event={"ID":"faa181ab-d548-4a6e-8aa4-ffe4189484a3","Type":"ContainerStarted","Data":"1a10a5544320a4aa992598068cfcb27213bb7cf946ac69750b67ddce901859de"} Nov 01 21:00:01.653781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:01.653734 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7304-eb614d"] Nov 01 21:00:01.695463 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:01.695390 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7304-eb614d"] Nov 01 21:00:02.103759 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.103722 4477 generic.go:356] "Generic (PLEG): container finished" podID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerID="ae5beec340516e30bc10bc1c6a381c12f1866992586bf2146bf3baa57960bec3" exitCode=60 Nov 01 21:00:02.103946 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.103813 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" event={"ID":"faa181ab-d548-4a6e-8aa4-ffe4189484a3","Type":"ContainerDied","Data":"ae5beec340516e30bc10bc1c6a381c12f1866992586bf2146bf3baa57960bec3"} Nov 01 21:00:02.104178 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.104154 4477 scope.go:122] "RemoveContainer" containerID="ae5beec340516e30bc10bc1c6a381c12f1866992586bf2146bf3baa57960bec3" Nov 01 21:00:02.104270 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.104171 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7304-eb614d" podUID="b2888814-e8f2-48bb-acd9-5db0d227232b" containerName="test-curl-7304-eb614d" containerID="cri-o://f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc" gracePeriod=2 Nov 01 21:00:02.266665 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:02.266583 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7304-eb614d\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="b2888814-e8f2-48bb-acd9-5db0d227232b" pod="opendatahub/test-curl-7304-eb614d" Nov 01 21:00:02.804160 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.803482 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9"] Nov 01 21:00:02.804680 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.804418 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b2888814-e8f2-48bb-acd9-5db0d227232b" containerName="test-curl-7304-eb614d" Nov 01 21:00:02.804680 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.804438 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2888814-e8f2-48bb-acd9-5db0d227232b" containerName="test-curl-7304-eb614d" Nov 01 21:00:02.804680 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.804548 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2888814-e8f2-48bb-acd9-5db0d227232b" containerName="test-curl-7304-eb614d" Nov 01 21:00:02.809186 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.809162 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:02.866267 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.866236 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9"] Nov 01 21:00:02.884195 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.883924 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:00:02.920629 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.920592 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:02.920820 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.920653 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:02.920820 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:02.920741 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnt72\" (UniqueName: \"kubernetes.io/projected/286ccd45-4a2a-4ed7-b424-45e1917569fc-kube-api-access-tnt72\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:03.022049 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.022010 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt72\" (UniqueName: \"kubernetes.io/projected/286ccd45-4a2a-4ed7-b424-45e1917569fc-kube-api-access-tnt72\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:03.022336 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.022305 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:03.022494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.022353 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:03.023200 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.023133 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:03.024954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.024929 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:03.079634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.079596 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnt72\" (UniqueName: \"kubernetes.io/projected/286ccd45-4a2a-4ed7-b424-45e1917569fc-kube-api-access-tnt72\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:03.115008 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.114957 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" event={"ID":"faa181ab-d548-4a6e-8aa4-ffe4189484a3","Type":"ContainerStarted","Data":"166f547f4aa145c6d4504efd0908664093c2d7b9585e9247b49d6b282c33a47d"} Nov 01 21:00:03.130723 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.130687 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:03.239857 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.238947 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" podStartSLOduration=3.238925853 podStartE2EDuration="3.238925853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:03.228370931 +0000 UTC m=+1756.878422487" watchObservedRunningTime="2026-11-01 21:00:03.238925853 +0000 UTC m=+1756.888977399" Nov 01 21:00:03.548401 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.548201 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9"] Nov 01 21:00:03.555605 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:00:03.555562 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286ccd45_4a2a_4ed7_b424_45e1917569fc.slice/crio-d510494c19cb4558de89ae911a13e507e45b4b3a2b578ab03f5cbf4f1102ed38 WatchSource:0}: Error finding container d510494c19cb4558de89ae911a13e507e45b4b3a2b578ab03f5cbf4f1102ed38: Status 404 returned error can't find the container with id d510494c19cb4558de89ae911a13e507e45b4b3a2b578ab03f5cbf4f1102ed38 Nov 01 21:00:03.838687 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.838585 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 21:00:03.937495 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.937471 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944061 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-token\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-token\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944165 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-socket\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944233 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-data\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944384 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-credential-socket\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944422 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-certs\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944461 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-kube-api-access-hnxjx\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-kube-api-access-hnxjx\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944545 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-envoy\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944573 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istiod-ca-cert\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.946128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.944719 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-podinfo\") pod \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\" (UID: \"3f79c420-64c0-464f-b08f-ceeef1b31aa5\") " Nov 01 21:00:03.962538 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.953713 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-credential-socket" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:03.971118 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.968271 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-token" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:03.971118 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.968503 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-socket" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:03.971118 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.968643 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-data" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:03.971622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.971401 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:03.971622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.971431 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:03.971622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.971446 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:03.975650 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.972859 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istiod-ca-cert" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:03.975650 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.973116 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-certs" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:03.975650 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.973162 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:03.984218 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.983554 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-envoy" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:03.984218 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.983829 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-kube-api-access-hnxjx" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "kube-api-access-hnxjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:03.985313 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:03.985282 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-podinfo" pod "3f79c420-64c0-464f-b08f-ceeef1b31aa5" (UID: "3f79c420-64c0-464f-b08f-ceeef1b31aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:00:04.074580 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.074541 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-podinfo\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.074733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.074610 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b188ab-d523-42b9-b369-c3b2944d42d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b188ab-d523-42b9-b369-c3b2944d42d6-istiod-ca-cert\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.074784 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.074726 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-certs\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.074835 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.074813 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-token\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-token\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.074901 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.074865 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-data\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.075266 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.075207 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-certs" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.076304 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.076262 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b188ab-d523-42b9-b369-c3b2944d42d6-istiod-ca-cert" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:04.076718 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.076662 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-credential-socket\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.076991 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-kube-api-access-lg6jj\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-kube-api-access-lg6jj\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.077231 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-envoy\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.077265 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-socket\") pod \"e7b188ab-d523-42b9-b369-c3b2944d42d6\" (UID: \"e7b188ab-d523-42b9-b369-c3b2944d42d6\") " Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.077933 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b188ab-d523-42b9-b369-c3b2944d42d6-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.077954 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.077968 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.077982 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnxjx\" (UniqueName: \"kubernetes.io/projected/3f79c420-64c0-464f-b08f-ceeef1b31aa5-kube-api-access-hnxjx\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.077996 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.078010 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.078022 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f79c420-64c0-464f-b08f-ceeef1b31aa5-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.078123 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-podinfo" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.078527 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-data" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.078985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.078788 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-credential-socket" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.080524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.080485 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-socket" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.082444 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.082419 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 21:00:04.083558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.083535 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-kube-api-access-lg6jj" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "kube-api-access-lg6jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:04.083674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.083658 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-token" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:04.085436 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.085416 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-envoy" pod "e7b188ab-d523-42b9-b369-c3b2944d42d6" (UID: "e7b188ab-d523-42b9-b369-c3b2944d42d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.170524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.170149 4477 generic.go:356] "Generic (PLEG): container finished" podID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerID="15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7" exitCode=0 Nov 01 21:00:04.170524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.170316 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" event={"ID":"e7b188ab-d523-42b9-b369-c3b2944d42d6","Type":"ContainerDied","Data":"15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7"} Nov 01 21:00:04.170524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.170435 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" event={"ID":"e7b188ab-d523-42b9-b369-c3b2944d42d6","Type":"ContainerDied","Data":"5ea1074b0460e69dfa9f3851fe197a6de5b4e2e17a9da93275e75dded9251042"} Nov 01 21:00:04.170524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.170480 4477 scope.go:122] "RemoveContainer" containerID="15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7" Nov 01 21:00:04.170729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.170700 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k" Nov 01 21:00:04.177281 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.172983 4477 generic.go:356] "Generic (PLEG): container finished" podID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerID="0688182a850706f0766858beba3d7658277e3e0161a3eecfa8029339121feaaf" exitCode=0 Nov 01 21:00:04.177281 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.173065 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" event={"ID":"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c","Type":"ContainerDied","Data":"0688182a850706f0766858beba3d7658277e3e0161a3eecfa8029339121feaaf"} Nov 01 21:00:04.177501 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.177473 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" event={"ID":"286ccd45-4a2a-4ed7-b424-45e1917569fc","Type":"ContainerStarted","Data":"049197af55fb5a101c2653090e58aa4f9f8bab71e64955db5db2ad67927340eb"} Nov 01 21:00:04.177682 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.177664 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" event={"ID":"286ccd45-4a2a-4ed7-b424-45e1917569fc","Type":"ContainerStarted","Data":"d510494c19cb4558de89ae911a13e507e45b4b3a2b578ab03f5cbf4f1102ed38"} Nov 01 21:00:04.180314 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.180293 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.180386 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.180320 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.180386 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.180337 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.180386 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.180358 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.180386 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.180372 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.180575 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.180387 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b188ab-d523-42b9-b369-c3b2944d42d6-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.180575 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.180403 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg6jj\" (UniqueName: \"kubernetes.io/projected/e7b188ab-d523-42b9-b369-c3b2944d42d6-kube-api-access-lg6jj\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.181221 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.181194 4477 generic.go:356] "Generic (PLEG): container finished" podID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerID="01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199" exitCode=0 Nov 01 21:00:04.181416 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.181254 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" Nov 01 21:00:04.181487 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.181270 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" event={"ID":"3f79c420-64c0-464f-b08f-ceeef1b31aa5","Type":"ContainerDied","Data":"01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199"} Nov 01 21:00:04.181575 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.181524 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg" event={"ID":"3f79c420-64c0-464f-b08f-ceeef1b31aa5","Type":"ContainerDied","Data":"19ccffa7604e3a4cd7b97b7a9ee38eb8000e65419ee524d595a89827a47c3a8f"} Nov 01 21:00:04.184178 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.184159 4477 generic.go:356] "Generic (PLEG): container finished" podID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerID="166f547f4aa145c6d4504efd0908664093c2d7b9585e9247b49d6b282c33a47d" exitCode=60 Nov 01 21:00:04.184277 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.184186 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" event={"ID":"faa181ab-d548-4a6e-8aa4-ffe4189484a3","Type":"ContainerDied","Data":"166f547f4aa145c6d4504efd0908664093c2d7b9585e9247b49d6b282c33a47d"} Nov 01 21:00:04.184471 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.184454 4477 scope.go:122] "RemoveContainer" containerID="166f547f4aa145c6d4504efd0908664093c2d7b9585e9247b49d6b282c33a47d" Nov 01 21:00:04.184727 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:04.184710 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5_odh-ai-gateway-infra(faa181ab-d548-4a6e-8aa4-ffe4189484a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" Nov 01 21:00:04.274894 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.274831 4477 scope.go:122] "RemoveContainer" containerID="15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7" Nov 01 21:00:04.276931 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:04.275377 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7\": container with ID starting with 15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7 not found: ID does not exist" containerID="15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7" Nov 01 21:00:04.276931 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.275411 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7"} err="failed to get container status \"15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7\": rpc error: code = NotFound desc = could not find container \"15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7\": container with ID starting with 15f97465ae8ff02ca3db93601275372d74f642234c8b9f75f63fffd9e5e537a7 not found: ID does not exist" Nov 01 21:00:04.276931 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.275435 4477 scope.go:122] "RemoveContainer" containerID="01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199" Nov 01 21:00:04.287914 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.287884 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 21:00:04.300493 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.300438 4477 scope.go:122] "RemoveContainer" containerID="01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199" Nov 01 21:00:04.301215 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:04.301020 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199\": container with ID starting with 01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199 not found: ID does not exist" containerID="01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199" Nov 01 21:00:04.301215 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.301060 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199"} err="failed to get container status \"01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199\": rpc error: code = NotFound desc = could not find container \"01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199\": container with ID starting with 01ccce53dd0aad01a60461f34abb270e4cb60f15831a9fbd294f8051c217a199 not found: ID does not exist" Nov 01 21:00:04.301215 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.301121 4477 scope.go:122] "RemoveContainer" containerID="ae5beec340516e30bc10bc1c6a381c12f1866992586bf2146bf3baa57960bec3" Nov 01 21:00:04.383759 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.383728 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istiod-ca-cert\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.383943 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.383798 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-podinfo\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.383943 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.383882 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-credential-socket\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.384066 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.383997 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-socket\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.384066 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.384044 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-token\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-token\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.384066 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.384094 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-kube-api-access-d97tm\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-kube-api-access-d97tm\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.384267 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.384158 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-data\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.384267 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.384185 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istiod-ca-cert" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:04.384267 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.384186 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-credential-socket" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.384267 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.384194 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-envoy\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.384518 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.384397 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-certs\") pod \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\" (UID: \"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c\") " Nov 01 21:00:04.384968 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.384910 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-data" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.385478 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.385182 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-certs" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.385478 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.385407 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.385478 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.385425 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.385478 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.385441 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.385478 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.385457 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.385897 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.385815 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-socket" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.387333 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.387301 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-envoy" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.387333 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.387317 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-podinfo" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:00:04.387477 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.387363 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-token" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:04.388057 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.388037 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-kube-api-access-d97tm" pod "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" (UID: "6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c"). InnerVolumeSpecName "kube-api-access-d97tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:04.400832 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.400781 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" podStartSLOduration=2.400731823 podStartE2EDuration="2.400731823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:04.396776742 +0000 UTC m=+1758.046828321" watchObservedRunningTime="2026-11-01 21:00:04.400731823 +0000 UTC m=+1758.050783367" Nov 01 21:00:04.423623 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.423591 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-eb614d" Nov 01 21:00:04.475919 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.475843 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 21:00:04.486865 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.486835 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-socket\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.487043 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.486877 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-data\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.487043 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.486919 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-certs\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.487234 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487155 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-socket" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.487298 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487232 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-data" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.487476 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487455 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-credential-socket" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.487559 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487513 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-certs" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.487602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.486957 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-credential-socket\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.487653 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487640 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2888814-e8f2-48bb-acd9-5db0d227232b-kube-api-access-55twj\" (UniqueName: \"kubernetes.io/projected/b2888814-e8f2-48bb-acd9-5db0d227232b-kube-api-access-55twj\") pod \"b2888814-e8f2-48bb-acd9-5db0d227232b\" (UID: \"b2888814-e8f2-48bb-acd9-5db0d227232b\") " Nov 01 21:00:04.487709 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487698 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e03d930-b982-4d8d-a611-31fdb427e38e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e03d930-b982-4d8d-a611-31fdb427e38e-istiod-ca-cert\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.487774 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487743 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-podinfo\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.487835 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487823 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-kube-api-access-x2wg2\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-kube-api-access-x2wg2\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487946 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-envoy\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.487983 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-token\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-token\") pod \"3e03d930-b982-4d8d-a611-31fdb427e38e\" (UID: \"3e03d930-b982-4d8d-a611-31fdb427e38e\") " Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.488035 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e03d930-b982-4d8d-a611-31fdb427e38e-istiod-ca-cert" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.488914 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.488938 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e03d930-b982-4d8d-a611-31fdb427e38e-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.488956 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.488981 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.488995 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.489010 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.489025 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.489038 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.489053 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.489830 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.489067 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97tm\" (UniqueName: \"kubernetes.io/projected/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c-kube-api-access-d97tm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.490526 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.490496 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-envoy" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:04.490767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.490747 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-kube-api-access-x2wg2" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "kube-api-access-x2wg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:04.490860 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.490773 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2888814-e8f2-48bb-acd9-5db0d227232b-kube-api-access-55twj" pod "b2888814-e8f2-48bb-acd9-5db0d227232b" (UID: "b2888814-e8f2-48bb-acd9-5db0d227232b"). InnerVolumeSpecName "kube-api-access-55twj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:04.490860 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.490791 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-podinfo" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:00:04.491866 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.491848 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-token" pod "3e03d930-b982-4d8d-a611-31fdb427e38e" (UID: "3e03d930-b982-4d8d-a611-31fdb427e38e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:04.507431 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.507402 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2888814-e8f2-48bb-acd9-5db0d227232b" path="/var/lib/kubelet/pods/b2888814-e8f2-48bb-acd9-5db0d227232b/volumes" Nov 01 21:00:04.591578 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.591551 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2wg2\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-kube-api-access-x2wg2\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.591578 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.591574 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.591578 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.591584 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.591795 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.591594 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55twj\" (UniqueName: \"kubernetes.io/projected/b2888814-e8f2-48bb-acd9-5db0d227232b-kube-api-access-55twj\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.591795 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.591604 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e03d930-b982-4d8d-a611-31fdb427e38e-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:04.795138 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.795105 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg"] Nov 01 21:00:04.801665 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.801631 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-dhjgg"] Nov 01 21:00:04.907247 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.907215 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k"] Nov 01 21:00:04.912223 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:04.912199 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-v5s7k"] Nov 01 21:00:05.193461 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.193426 4477 generic.go:356] "Generic (PLEG): container finished" podID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerID="12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b" exitCode=0 Nov 01 21:00:05.193648 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.193490 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" Nov 01 21:00:05.193648 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.193506 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" event={"ID":"3e03d930-b982-4d8d-a611-31fdb427e38e","Type":"ContainerDied","Data":"12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b"} Nov 01 21:00:05.193648 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.193575 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2" event={"ID":"3e03d930-b982-4d8d-a611-31fdb427e38e","Type":"ContainerDied","Data":"3a0ca72645d432935650d6084e8d996a2edbe4cd7bdeb29ec82c1d7aae51b13d"} Nov 01 21:00:05.193648 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.193602 4477 scope.go:122] "RemoveContainer" containerID="12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b" Nov 01 21:00:05.195766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.195741 4477 scope.go:122] "RemoveContainer" containerID="166f547f4aa145c6d4504efd0908664093c2d7b9585e9247b49d6b282c33a47d" Nov 01 21:00:05.196114 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:05.196088 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5_odh-ai-gateway-infra(faa181ab-d548-4a6e-8aa4-ffe4189484a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" Nov 01 21:00:05.197806 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.197783 4477 generic.go:356] "Generic (PLEG): container finished" podID="b2888814-e8f2-48bb-acd9-5db0d227232b" containerID="f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc" exitCode=137 Nov 01 21:00:05.197905 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.197815 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7304-eb614d" Nov 01 21:00:05.201046 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.201020 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" Nov 01 21:00:05.201188 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.201138 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6" event={"ID":"6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c","Type":"ContainerDied","Data":"54cdf7bd902cca82ab1095899f34b1ffa5c74d9654324cc96a0fd9fdce08f91e"} Nov 01 21:00:05.202916 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.202893 4477 generic.go:356] "Generic (PLEG): container finished" podID="286ccd45-4a2a-4ed7-b424-45e1917569fc" containerID="049197af55fb5a101c2653090e58aa4f9f8bab71e64955db5db2ad67927340eb" exitCode=0 Nov 01 21:00:05.203031 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.203011 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" event={"ID":"286ccd45-4a2a-4ed7-b424-45e1917569fc","Type":"ContainerDied","Data":"049197af55fb5a101c2653090e58aa4f9f8bab71e64955db5db2ad67927340eb"} Nov 01 21:00:05.208671 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.208651 4477 scope.go:122] "RemoveContainer" containerID="12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b" Nov 01 21:00:05.208958 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:05.208933 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b\": container with ID starting with 12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b not found: ID does not exist" containerID="12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b" Nov 01 21:00:05.209014 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.208969 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b"} err="failed to get container status \"12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b\": rpc error: code = NotFound desc = could not find container \"12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b\": container with ID starting with 12ab5a4fb84904f8a0093b4b811320c350a65159fa1b935735eb1278d4a65b9b not found: ID does not exist" Nov 01 21:00:05.209014 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.208991 4477 scope.go:122] "RemoveContainer" containerID="f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc" Nov 01 21:00:05.219626 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.219606 4477 scope.go:122] "RemoveContainer" containerID="f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc" Nov 01 21:00:05.219926 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:05.219907 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc\": container with ID starting with f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc not found: ID does not exist" containerID="f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc" Nov 01 21:00:05.219998 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.219931 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc"} err="failed to get container status \"f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc\": rpc error: code = NotFound desc = could not find container \"f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc\": container with ID starting with f35cb4893d643876b5258843373bd6d0999adbe46809d67cb2631fab647e0dfc not found: ID does not exist" Nov 01 21:00:05.219998 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.219946 4477 scope.go:122] "RemoveContainer" containerID="0688182a850706f0766858beba3d7658277e3e0161a3eecfa8029339121feaaf" Nov 01 21:00:05.365235 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.365200 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2"] Nov 01 21:00:05.396037 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.396004 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-mwvv2"] Nov 01 21:00:05.593157 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.593123 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6"] Nov 01 21:00:05.596814 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:05.596791 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-kdwp6"] Nov 01 21:00:06.416905 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.416878 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:06.507014 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.506929 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" path="/var/lib/kubelet/pods/3e03d930-b982-4d8d-a611-31fdb427e38e/volumes" Nov 01 21:00:06.507679 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.507657 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" path="/var/lib/kubelet/pods/3f79c420-64c0-464f-b08f-ceeef1b31aa5/volumes" Nov 01 21:00:06.508336 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.508316 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" path="/var/lib/kubelet/pods/6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c/volumes" Nov 01 21:00:06.509136 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.509121 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" path="/var/lib/kubelet/pods/e7b188ab-d523-42b9-b369-c3b2944d42d6/volumes" Nov 01 21:00:06.518468 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.518441 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-openshift-ca\") pod \"286ccd45-4a2a-4ed7-b424-45e1917569fc\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " Nov 01 21:00:06.518643 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.518478 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286ccd45-4a2a-4ed7-b424-45e1917569fc-kube-api-access-tnt72\" (UniqueName: \"kubernetes.io/projected/286ccd45-4a2a-4ed7-b424-45e1917569fc-kube-api-access-tnt72\") pod \"286ccd45-4a2a-4ed7-b424-45e1917569fc\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " Nov 01 21:00:06.518720 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.518665 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-xks-ca\") pod \"286ccd45-4a2a-4ed7-b424-45e1917569fc\" (UID: \"286ccd45-4a2a-4ed7-b424-45e1917569fc\") " Nov 01 21:00:06.518776 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.518753 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-openshift-ca" pod "286ccd45-4a2a-4ed7-b424-45e1917569fc" (UID: "286ccd45-4a2a-4ed7-b424-45e1917569fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:06.518988 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.518973 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-openshift-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.520829 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.520802 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286ccd45-4a2a-4ed7-b424-45e1917569fc-kube-api-access-tnt72" pod "286ccd45-4a2a-4ed7-b424-45e1917569fc" (UID: "286ccd45-4a2a-4ed7-b424-45e1917569fc"). InnerVolumeSpecName "kube-api-access-tnt72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:06.520963 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.520827 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-xks-ca" pod "286ccd45-4a2a-4ed7-b424-45e1917569fc" (UID: "286ccd45-4a2a-4ed7-b424-45e1917569fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:00:06.566724 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:06.566674 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:00:06.621667 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.621623 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnt72\" (UniqueName: \"kubernetes.io/projected/286ccd45-4a2a-4ed7-b424-45e1917569fc-kube-api-access-tnt72\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.621667 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:06.621668 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286ccd45-4a2a-4ed7-b424-45e1917569fc-maas-api-xks-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:07.233979 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:07.233187 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" event={"ID":"286ccd45-4a2a-4ed7-b424-45e1917569fc","Type":"ContainerDied","Data":"d510494c19cb4558de89ae911a13e507e45b4b3a2b578ab03f5cbf4f1102ed38"} Nov 01 21:00:07.233979 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:07.233296 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d510494c19cb4558de89ae911a13e507e45b4b3a2b578ab03f5cbf4f1102ed38" Nov 01 21:00:07.233979 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:07.233384 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9" Nov 01 21:00:15.497411 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:15.497380 4477 scope.go:122] "RemoveContainer" containerID="166f547f4aa145c6d4504efd0908664093c2d7b9585e9247b49d6b282c33a47d" Nov 01 21:00:16.315729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:16.315685 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" event={"ID":"faa181ab-d548-4a6e-8aa4-ffe4189484a3","Type":"ContainerStarted","Data":"a5deffffda99bb894099a743e02dfd5dad73e06fc6b0348df9b164db058a76d2"} Nov 01 21:00:17.337090 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:17.337039 4477 generic.go:356] "Generic (PLEG): container finished" podID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerID="a5deffffda99bb894099a743e02dfd5dad73e06fc6b0348df9b164db058a76d2" exitCode=60 Nov 01 21:00:17.337817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:17.337107 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" event={"ID":"faa181ab-d548-4a6e-8aa4-ffe4189484a3","Type":"ContainerDied","Data":"a5deffffda99bb894099a743e02dfd5dad73e06fc6b0348df9b164db058a76d2"} Nov 01 21:00:17.337817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:17.337173 4477 scope.go:122] "RemoveContainer" containerID="166f547f4aa145c6d4504efd0908664093c2d7b9585e9247b49d6b282c33a47d" Nov 01 21:00:17.337817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:17.337578 4477 scope.go:122] "RemoveContainer" containerID="a5deffffda99bb894099a743e02dfd5dad73e06fc6b0348df9b164db058a76d2" Nov 01 21:00:17.338096 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:17.337889 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5_odh-ai-gateway-infra(faa181ab-d548-4a6e-8aa4-ffe4189484a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" Nov 01 21:00:17.444118 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:17.444044 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5"] Nov 01 21:00:18.555757 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:18.555729 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:18.624681 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:18.624579 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa181ab-d548-4a6e-8aa4-ffe4189484a3-kube-api-access-tfvjg\" (UniqueName: \"kubernetes.io/projected/faa181ab-d548-4a6e-8aa4-ffe4189484a3-kube-api-access-tfvjg\") pod \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\" (UID: \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\") " Nov 01 21:00:18.624681 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:18.624665 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa181ab-d548-4a6e-8aa4-ffe4189484a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa181ab-d548-4a6e-8aa4-ffe4189484a3-maas-api-service-ca\") pod \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\" (UID: \"faa181ab-d548-4a6e-8aa4-ffe4189484a3\") " Nov 01 21:00:18.625028 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:18.625006 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa181ab-d548-4a6e-8aa4-ffe4189484a3-maas-api-service-ca" pod "faa181ab-d548-4a6e-8aa4-ffe4189484a3" (UID: "faa181ab-d548-4a6e-8aa4-ffe4189484a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:18.625380 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:18.625356 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa181ab-d548-4a6e-8aa4-ffe4189484a3-maas-api-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:18.627167 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:18.627143 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa181ab-d548-4a6e-8aa4-ffe4189484a3-kube-api-access-tfvjg" pod "faa181ab-d548-4a6e-8aa4-ffe4189484a3" (UID: "faa181ab-d548-4a6e-8aa4-ffe4189484a3"). InnerVolumeSpecName "kube-api-access-tfvjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:18.727185 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:18.727139 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvjg\" (UniqueName: \"kubernetes.io/projected/faa181ab-d548-4a6e-8aa4-ffe4189484a3-kube-api-access-tfvjg\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:19.357403 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:19.357373 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" Nov 01 21:00:19.357619 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:19.357418 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5" event={"ID":"faa181ab-d548-4a6e-8aa4-ffe4189484a3","Type":"ContainerDied","Data":"1a10a5544320a4aa992598068cfcb27213bb7cf946ac69750b67ddce901859de"} Nov 01 21:00:19.357619 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:19.357480 4477 scope.go:122] "RemoveContainer" containerID="a5deffffda99bb894099a743e02dfd5dad73e06fc6b0348df9b164db058a76d2" Nov 01 21:00:19.526657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:19.526612 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5"] Nov 01 21:00:19.692739 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:19.689832 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9b439-29892780-8h2h5"] Nov 01 21:00:20.505085 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:20.505025 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" path="/var/lib/kubelet/pods/faa181ab-d548-4a6e-8aa4-ffe4189484a3/volumes" Nov 01 21:00:35.436003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:00:35.435971 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:00:36.556562 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:00:36.556520 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:01:06.556836 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:06.556794 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:01:22.678151 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678116 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7295-3e3319"] Nov 01 21:01:22.678701 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678689 4477 cpu_manager.go:414] "containerMap: removing container" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" Nov 01 21:01:22.678767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678703 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" Nov 01 21:01:22.678767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678721 4477 cpu_manager.go:414] "containerMap: removing container" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" Nov 01 21:01:22.678767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678726 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" Nov 01 21:01:22.678767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678739 4477 cpu_manager.go:414] "containerMap: removing container" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.678767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678745 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.678767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678750 4477 cpu_manager.go:414] "containerMap: removing container" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.678767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678755 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.678767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678768 4477 cpu_manager.go:414] "containerMap: removing container" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678773 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678785 4477 cpu_manager.go:414] "containerMap: removing container" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678790 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678803 4477 cpu_manager.go:414] "containerMap: removing container" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678807 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678813 4477 cpu_manager.go:414] "containerMap: removing container" podUID="286ccd45-4a2a-4ed7-b424-45e1917569fc" containerName="revoke-keys" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678817 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286ccd45-4a2a-4ed7-b424-45e1917569fc" containerName="revoke-keys" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678918 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678926 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678934 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd69fd9-9a03-42be-b4d5-6ab1513cfd5c" containerName="istio-proxy" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678941 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa181ab-d548-4a6e-8aa4-ffe4189484a3" containerName="cleanup" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678949 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b188ab-d523-42b9-b369-c3b2944d42d6" containerName="istio-proxy" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678954 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="286ccd45-4a2a-4ed7-b424-45e1917569fc" containerName="revoke-keys" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678961 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e03d930-b982-4d8d-a611-31fdb427e38e" containerName="istio-proxy" Nov 01 21:01:22.679013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.678968 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f79c420-64c0-464f-b08f-ceeef1b31aa5" containerName="istio-proxy" Nov 01 21:01:22.682687 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.682672 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:22.734839 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.734815 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-3e3319"] Nov 01 21:01:22.744553 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.744522 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhcbb\" (UniqueName: \"kubernetes.io/projected/ecfe229d-78a8-40cf-b920-932944a1c4b7-kube-api-access-hhcbb\") pod \"test-curl-7295-3e3319\" (UID: \"ecfe229d-78a8-40cf-b920-932944a1c4b7\") " pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:22.779496 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.779472 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8mhs\"" Nov 01 21:01:22.845762 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.845727 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhcbb\" (UniqueName: \"kubernetes.io/projected/ecfe229d-78a8-40cf-b920-932944a1c4b7-kube-api-access-hhcbb\") pod \"test-curl-7295-3e3319\" (UID: \"ecfe229d-78a8-40cf-b920-932944a1c4b7\") " pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:22.901970 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.901940 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhcbb\" (UniqueName: \"kubernetes.io/projected/ecfe229d-78a8-40cf-b920-932944a1c4b7-kube-api-access-hhcbb\") pod \"test-curl-7295-3e3319\" (UID: \"ecfe229d-78a8-40cf-b920-932944a1c4b7\") " pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:22.995456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:22.995370 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:23.385942 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:23.385910 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-3e3319"] Nov 01 21:01:23.387541 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:01:23.387504 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecfe229d_78a8_40cf_b920_932944a1c4b7.slice/crio-689dce134ee4cb09ad1b760b2458fbf1228a170ae95c7599fabdf92813412157 WatchSource:0}: Error finding container 689dce134ee4cb09ad1b760b2458fbf1228a170ae95c7599fabdf92813412157: Status 404 returned error can't find the container with id 689dce134ee4cb09ad1b760b2458fbf1228a170ae95c7599fabdf92813412157 Nov 01 21:01:23.870893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:23.870864 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-3e3319" event={"ID":"ecfe229d-78a8-40cf-b920-932944a1c4b7","Type":"ContainerStarted","Data":"d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21"} Nov 01 21:01:23.871326 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:23.870904 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-3e3319" event={"ID":"ecfe229d-78a8-40cf-b920-932944a1c4b7","Type":"ContainerStarted","Data":"689dce134ee4cb09ad1b760b2458fbf1228a170ae95c7599fabdf92813412157"} Nov 01 21:01:23.986494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:23.986431 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7295-3e3319" podStartSLOduration=1.9864159049999999 podStartE2EDuration="1.986415905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:23.981286337 +0000 UTC m=+1837.631337882" watchObservedRunningTime="2026-11-01 21:01:23.986415905 +0000 UTC m=+1837.636467449" Nov 01 21:01:25.320247 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.320207 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7295-3e3319"] Nov 01 21:01:25.326353 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.326320 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7295-3e3319"] Nov 01 21:01:25.457746 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.457711 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7295-c340ab"] Nov 01 21:01:25.458688 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.458664 4477 cpu_manager.go:414] "containerMap: removing container" podUID="ecfe229d-78a8-40cf-b920-932944a1c4b7" containerName="test-curl-7295-3e3319" Nov 01 21:01:25.458749 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.458692 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfe229d-78a8-40cf-b920-932944a1c4b7" containerName="test-curl-7295-3e3319" Nov 01 21:01:25.458911 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.458899 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfe229d-78a8-40cf-b920-932944a1c4b7" containerName="test-curl-7295-3e3319" Nov 01 21:01:25.463020 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.462990 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-c340ab" Nov 01 21:01:25.512634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.512605 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-c340ab"] Nov 01 21:01:25.582437 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.582345 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hcbv\" (UniqueName: \"kubernetes.io/projected/a4e33796-bb6b-4feb-966e-b2a7f53ea021-kube-api-access-4hcbv\") pod \"test-curl-7295-c340ab\" (UID: \"a4e33796-bb6b-4feb-966e-b2a7f53ea021\") " pod="opendatahub/test-curl-7295-c340ab" Nov 01 21:01:25.683629 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.683599 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hcbv\" (UniqueName: \"kubernetes.io/projected/a4e33796-bb6b-4feb-966e-b2a7f53ea021-kube-api-access-4hcbv\") pod \"test-curl-7295-c340ab\" (UID: \"a4e33796-bb6b-4feb-966e-b2a7f53ea021\") " pod="opendatahub/test-curl-7295-c340ab" Nov 01 21:01:25.740563 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.740533 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hcbv\" (UniqueName: \"kubernetes.io/projected/a4e33796-bb6b-4feb-966e-b2a7f53ea021-kube-api-access-4hcbv\") pod \"test-curl-7295-c340ab\" (UID: \"a4e33796-bb6b-4feb-966e-b2a7f53ea021\") " pod="opendatahub/test-curl-7295-c340ab" Nov 01 21:01:25.775190 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.775166 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-c340ab" Nov 01 21:01:25.888228 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:25.888164 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7295-3e3319" podUID="ecfe229d-78a8-40cf-b920-932944a1c4b7" containerName="test-curl-7295-3e3319" containerID="cri-o://d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21" gracePeriod=2 Nov 01 21:01:25.939218 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:25.939173 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-3e3319\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="ecfe229d-78a8-40cf-b920-932944a1c4b7" pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:26.152144 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:26.152112 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-c340ab"] Nov 01 21:01:26.153309 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:01:26.153277 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e33796_bb6b_4feb_966e_b2a7f53ea021.slice/crio-9aa69ff59117575b59ea855b527a5ff318c7f5bef6666bfe09ab315b50e444d4 WatchSource:0}: Error finding container 9aa69ff59117575b59ea855b527a5ff318c7f5bef6666bfe09ab315b50e444d4: Status 404 returned error can't find the container with id 9aa69ff59117575b59ea855b527a5ff318c7f5bef6666bfe09ab315b50e444d4 Nov 01 21:01:26.552596 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:26.552485 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-3e3319\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="ecfe229d-78a8-40cf-b920-932944a1c4b7" pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:26.896137 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:26.896091 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-c340ab" event={"ID":"a4e33796-bb6b-4feb-966e-b2a7f53ea021","Type":"ContainerStarted","Data":"e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d"} Nov 01 21:01:26.896137 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:26.896143 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-c340ab" event={"ID":"a4e33796-bb6b-4feb-966e-b2a7f53ea021","Type":"ContainerStarted","Data":"9aa69ff59117575b59ea855b527a5ff318c7f5bef6666bfe09ab315b50e444d4"} Nov 01 21:01:27.049755 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:27.049701 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7295-c340ab" podStartSLOduration=2.049686932 podStartE2EDuration="2.049686932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:27.047512886 +0000 UTC m=+1840.697564432" watchObservedRunningTime="2026-11-01 21:01:27.049686932 +0000 UTC m=+1840.699738476" Nov 01 21:01:28.153243 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.153176 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:28.316158 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.316125 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfe229d-78a8-40cf-b920-932944a1c4b7-kube-api-access-hhcbb\" (UniqueName: \"kubernetes.io/projected/ecfe229d-78a8-40cf-b920-932944a1c4b7-kube-api-access-hhcbb\") pod \"ecfe229d-78a8-40cf-b920-932944a1c4b7\" (UID: \"ecfe229d-78a8-40cf-b920-932944a1c4b7\") " Nov 01 21:01:28.318521 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.318488 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfe229d-78a8-40cf-b920-932944a1c4b7-kube-api-access-hhcbb" pod "ecfe229d-78a8-40cf-b920-932944a1c4b7" (UID: "ecfe229d-78a8-40cf-b920-932944a1c4b7"). InnerVolumeSpecName "kube-api-access-hhcbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:01:28.418520 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.418424 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhcbb\" (UniqueName: \"kubernetes.io/projected/ecfe229d-78a8-40cf-b920-932944a1c4b7-kube-api-access-hhcbb\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:28.451306 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.451273 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7295-c340ab"] Nov 01 21:01:28.457332 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.457297 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7295-c340ab"] Nov 01 21:01:28.507770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.507727 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfe229d-78a8-40cf-b920-932944a1c4b7" path="/var/lib/kubelet/pods/ecfe229d-78a8-40cf-b920-932944a1c4b7/volumes" Nov 01 21:01:28.589791 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.589755 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7295-3f96a3"] Nov 01 21:01:28.590385 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.590372 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a4e33796-bb6b-4feb-966e-b2a7f53ea021" containerName="test-curl-7295-c340ab" Nov 01 21:01:28.590458 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.590387 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e33796-bb6b-4feb-966e-b2a7f53ea021" containerName="test-curl-7295-c340ab" Nov 01 21:01:28.590536 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.590526 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e33796-bb6b-4feb-966e-b2a7f53ea021" containerName="test-curl-7295-c340ab" Nov 01 21:01:28.594125 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.594106 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:28.647895 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.647800 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-3f96a3"] Nov 01 21:01:28.661778 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.661740 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h"] Nov 01 21:01:28.666944 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.666919 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.718710 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.718683 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-29baa8-serving-cert\"" Nov 01 21:01:28.721148 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.721123 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h"] Nov 01 21:01:28.722129 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.722102 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhjg\" (UniqueName: \"kubernetes.io/projected/bed90ad2-bb1f-46fe-8e7c-b103d4949dc8-kube-api-access-fdhjg\") pod \"test-curl-7295-3f96a3\" (UID: \"bed90ad2-bb1f-46fe-8e7c-b103d4949dc8\") " pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:28.823689 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.823656 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.823916 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.823882 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhjg\" (UniqueName: \"kubernetes.io/projected/bed90ad2-bb1f-46fe-8e7c-b103d4949dc8-kube-api-access-fdhjg\") pod \"test-curl-7295-3f96a3\" (UID: \"bed90ad2-bb1f-46fe-8e7c-b103d4949dc8\") " pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:28.823994 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.823928 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.824091 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.824038 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgfhl\" (UniqueName: \"kubernetes.io/projected/8fb70160-bb70-4ebf-b279-8f792298d96b-kube-api-access-qgfhl\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.824230 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.824207 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb70160-bb70-4ebf-b279-8f792298d96b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.880154 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.880030 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhjg\" (UniqueName: \"kubernetes.io/projected/bed90ad2-bb1f-46fe-8e7c-b103d4949dc8-kube-api-access-fdhjg\") pod \"test-curl-7295-3f96a3\" (UID: \"bed90ad2-bb1f-46fe-8e7c-b103d4949dc8\") " pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:28.909194 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.909154 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:28.925729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.922820 4477 generic.go:356] "Generic (PLEG): container finished" podID="ecfe229d-78a8-40cf-b920-932944a1c4b7" containerID="d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21" exitCode=137 Nov 01 21:01:28.925729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.923268 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7295-c340ab" podUID="a4e33796-bb6b-4feb-966e-b2a7f53ea021" containerName="test-curl-7295-c340ab" containerID="cri-o://e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d" gracePeriod=2 Nov 01 21:01:28.925729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.923416 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-3e3319" Nov 01 21:01:28.925729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.923935 4477 scope.go:122] "RemoveContainer" containerID="d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21" Nov 01 21:01:28.925729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.925234 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.925729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.925272 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgfhl\" (UniqueName: \"kubernetes.io/projected/8fb70160-bb70-4ebf-b279-8f792298d96b-kube-api-access-qgfhl\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.925729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.925376 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb70160-bb70-4ebf-b279-8f792298d96b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.925729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.925426 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.926779 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.926350 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb70160-bb70-4ebf-b279-8f792298d96b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.929124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.929093 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.929261 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.929177 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.970974 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.970597 4477 scope.go:122] "RemoveContainer" containerID="d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21" Nov 01 21:01:28.971132 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:28.971037 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21\": container with ID starting with d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21 not found: ID does not exist" containerID="d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21" Nov 01 21:01:28.971219 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.971106 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21"} err="failed to get container status \"d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21\": rpc error: code = NotFound desc = could not find container \"d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21\": container with ID starting with d03acbd56c5d41fbc2c4d2aacf3c846e8a591fd5154e6f2982e2f1def2819c21 not found: ID does not exist" Nov 01 21:01:28.983929 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.983898 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgfhl\" (UniqueName: \"kubernetes.io/projected/8fb70160-bb70-4ebf-b279-8f792298d96b-kube-api-access-qgfhl\") pod \"maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:28.986967 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:28.986938 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:29.308884 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:29.308859 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-3f96a3"] Nov 01 21:01:29.309686 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:01:29.309661 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbed90ad2_bb1f_46fe_8e7c_b103d4949dc8.slice/crio-8daca7a2fbe3c3397aa42151aea638915f167a7ad637bc273bfde3108548cbab WatchSource:0}: Error finding container 8daca7a2fbe3c3397aa42151aea638915f167a7ad637bc273bfde3108548cbab: Status 404 returned error can't find the container with id 8daca7a2fbe3c3397aa42151aea638915f167a7ad637bc273bfde3108548cbab Nov 01 21:01:29.424688 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:29.424659 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h"] Nov 01 21:01:29.425145 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:01:29.425116 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb70160_bb70_4ebf_b279_8f792298d96b.slice/crio-1e50bf719665d2764c03f3bcedf410ee7ad23c34577ba6e6635c632b39bcfe4d WatchSource:0}: Error finding container 1e50bf719665d2764c03f3bcedf410ee7ad23c34577ba6e6635c632b39bcfe4d: Status 404 returned error can't find the container with id 1e50bf719665d2764c03f3bcedf410ee7ad23c34577ba6e6635c632b39bcfe4d Nov 01 21:01:29.932491 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:29.932458 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-3f96a3" event={"ID":"bed90ad2-bb1f-46fe-8e7c-b103d4949dc8","Type":"ContainerStarted","Data":"d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171"} Nov 01 21:01:29.932690 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:29.932507 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-3f96a3" event={"ID":"bed90ad2-bb1f-46fe-8e7c-b103d4949dc8","Type":"ContainerStarted","Data":"8daca7a2fbe3c3397aa42151aea638915f167a7ad637bc273bfde3108548cbab"} Nov 01 21:01:29.935185 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:29.935157 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" event={"ID":"8fb70160-bb70-4ebf-b279-8f792298d96b","Type":"ContainerStarted","Data":"1e50bf719665d2764c03f3bcedf410ee7ad23c34577ba6e6635c632b39bcfe4d"} Nov 01 21:01:30.044539 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:30.044484 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7295-3f96a3" podStartSLOduration=2.044468687 podStartE2EDuration="2.044468687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:30.043469504 +0000 UTC m=+1843.693521071" watchObservedRunningTime="2026-11-01 21:01:30.044468687 +0000 UTC m=+1843.694520231" Nov 01 21:01:30.945616 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:30.945578 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" event={"ID":"8fb70160-bb70-4ebf-b279-8f792298d96b","Type":"ContainerStarted","Data":"fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5"} Nov 01 21:01:30.946193 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:30.946166 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:31.073837 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.073759 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" podStartSLOduration=2.234443319 podStartE2EDuration="3.073738822s" totalImagesPullingTime="839.295503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:31.062525538 +0000 UTC m=+1844.712577109" watchObservedRunningTime="2026-11-01 21:01:31.073738822 +0000 UTC m=+1844.723790370" Nov 01 21:01:31.401184 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.401156 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-c340ab" Nov 01 21:01:31.476921 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.476884 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7295-3f96a3"] Nov 01 21:01:31.506123 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.481699 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7295-3f96a3"] Nov 01 21:01:31.571082 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.571045 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e33796-bb6b-4feb-966e-b2a7f53ea021-kube-api-access-4hcbv\" (UniqueName: \"kubernetes.io/projected/a4e33796-bb6b-4feb-966e-b2a7f53ea021-kube-api-access-4hcbv\") pod \"a4e33796-bb6b-4feb-966e-b2a7f53ea021\" (UID: \"a4e33796-bb6b-4feb-966e-b2a7f53ea021\") " Nov 01 21:01:31.573385 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.573352 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e33796-bb6b-4feb-966e-b2a7f53ea021-kube-api-access-4hcbv" pod "a4e33796-bb6b-4feb-966e-b2a7f53ea021" (UID: "a4e33796-bb6b-4feb-966e-b2a7f53ea021"). InnerVolumeSpecName "kube-api-access-4hcbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:01:31.612767 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.612732 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7295-8eb8ff"] Nov 01 21:01:31.613582 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.613557 4477 cpu_manager.go:414] "containerMap: removing container" podUID="bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" containerName="test-curl-7295-3f96a3" Nov 01 21:01:31.613582 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.613580 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" containerName="test-curl-7295-3f96a3" Nov 01 21:01:31.613774 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.613692 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" containerName="test-curl-7295-3f96a3" Nov 01 21:01:31.617685 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.617659 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:31.671459 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.671421 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-8eb8ff"] Nov 01 21:01:31.674059 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.673702 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hcbv\" (UniqueName: \"kubernetes.io/projected/a4e33796-bb6b-4feb-966e-b2a7f53ea021-kube-api-access-4hcbv\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:31.775135 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.775089 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4fm\" (UniqueName: \"kubernetes.io/projected/bd31b427-10ae-4070-85d2-bec0d2869034-kube-api-access-wc4fm\") pod \"test-curl-7295-8eb8ff\" (UID: \"bd31b427-10ae-4070-85d2-bec0d2869034\") " pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:31.877496 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.877389 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4fm\" (UniqueName: \"kubernetes.io/projected/bd31b427-10ae-4070-85d2-bec0d2869034-kube-api-access-wc4fm\") pod \"test-curl-7295-8eb8ff\" (UID: \"bd31b427-10ae-4070-85d2-bec0d2869034\") " pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:31.933983 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.933954 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4fm\" (UniqueName: \"kubernetes.io/projected/bd31b427-10ae-4070-85d2-bec0d2869034-kube-api-access-wc4fm\") pod \"test-curl-7295-8eb8ff\" (UID: \"bd31b427-10ae-4070-85d2-bec0d2869034\") " pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:31.955406 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.955373 4477 generic.go:356] "Generic (PLEG): container finished" podID="a4e33796-bb6b-4feb-966e-b2a7f53ea021" containerID="e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d" exitCode=137 Nov 01 21:01:31.955887 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.955430 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-c340ab" Nov 01 21:01:31.955887 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.955445 4477 scope.go:122] "RemoveContainer" containerID="e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d" Nov 01 21:01:31.955887 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.955770 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7295-3f96a3" podUID="bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" containerName="test-curl-7295-3f96a3" containerID="cri-o://d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171" gracePeriod=2 Nov 01 21:01:31.976157 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.976133 4477 scope.go:122] "RemoveContainer" containerID="e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d" Nov 01 21:01:31.976510 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:31.976485 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d\": container with ID starting with e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d not found: ID does not exist" containerID="e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d" Nov 01 21:01:31.976571 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:31.976522 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d"} err="failed to get container status \"e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d\": rpc error: code = NotFound desc = could not find container \"e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d\": container with ID starting with e73060d1fb39fbe2e0f48dbe4d4aa3f018c637d2a92160c7a59d57d9906d3d6d not found: ID does not exist" Nov 01 21:01:32.006856 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:32.006806 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-3f96a3\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:32.057578 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:32.057535 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-c340ab\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="a4e33796-bb6b-4feb-966e-b2a7f53ea021" pod="opendatahub/test-curl-7295-c340ab" Nov 01 21:01:32.108205 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:32.108163 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-3f96a3\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:32.230795 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:32.230707 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:32.507282 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:32.507201 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e33796-bb6b-4feb-966e-b2a7f53ea021" path="/var/lib/kubelet/pods/a4e33796-bb6b-4feb-966e-b2a7f53ea021/volumes" Nov 01 21:01:32.702177 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:01:32.702133 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd31b427_10ae_4070_85d2_bec0d2869034.slice/crio-efc7255249a0f6d07a6f649c0f04fc6df77f963c3b5248298131cf331875f941 WatchSource:0}: Error finding container efc7255249a0f6d07a6f649c0f04fc6df77f963c3b5248298131cf331875f941: Status 404 returned error can't find the container with id efc7255249a0f6d07a6f649c0f04fc6df77f963c3b5248298131cf331875f941 Nov 01 21:01:32.702853 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:32.702827 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-8eb8ff"] Nov 01 21:01:32.965981 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:32.965941 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-8eb8ff" event={"ID":"bd31b427-10ae-4070-85d2-bec0d2869034","Type":"ContainerStarted","Data":"cfe2c746852f95a9c802c558119fb3c9f9419c403f2bf4f37d809e890f7997c8"} Nov 01 21:01:32.966472 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:32.965995 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-8eb8ff" event={"ID":"bd31b427-10ae-4070-85d2-bec0d2869034","Type":"ContainerStarted","Data":"efc7255249a0f6d07a6f649c0f04fc6df77f963c3b5248298131cf331875f941"} Nov 01 21:01:33.078344 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:33.078290 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7295-8eb8ff" podStartSLOduration=2.078275515 podStartE2EDuration="2.078275515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:33.076275069 +0000 UTC m=+1846.726326616" watchObservedRunningTime="2026-11-01 21:01:33.078275515 +0000 UTC m=+1846.728327059" Nov 01 21:01:34.245918 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:34.245891 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:34.323579 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:34.323547 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed90ad2-bb1f-46fe-8e7c-b103d4949dc8-kube-api-access-fdhjg\" (UniqueName: \"kubernetes.io/projected/bed90ad2-bb1f-46fe-8e7c-b103d4949dc8-kube-api-access-fdhjg\") pod \"bed90ad2-bb1f-46fe-8e7c-b103d4949dc8\" (UID: \"bed90ad2-bb1f-46fe-8e7c-b103d4949dc8\") " Nov 01 21:01:34.325936 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:34.325901 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed90ad2-bb1f-46fe-8e7c-b103d4949dc8-kube-api-access-fdhjg" pod "bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" (UID: "bed90ad2-bb1f-46fe-8e7c-b103d4949dc8"). InnerVolumeSpecName "kube-api-access-fdhjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:01:34.425617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:34.425587 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhjg\" (UniqueName: \"kubernetes.io/projected/bed90ad2-bb1f-46fe-8e7c-b103d4949dc8-kube-api-access-fdhjg\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:34.505607 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:34.505526 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" path="/var/lib/kubelet/pods/bed90ad2-bb1f-46fe-8e7c-b103d4949dc8/volumes" Nov 01 21:01:34.533942 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:34.533902 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7295-8eb8ff"] Nov 01 21:01:34.539574 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:34.539542 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7295-8eb8ff"] Nov 01 21:01:35.003291 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.003253 4477 generic.go:356] "Generic (PLEG): container finished" podID="bed90ad2-bb1f-46fe-8e7c-b103d4949dc8" containerID="d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171" exitCode=137 Nov 01 21:01:35.003457 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.003312 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-3f96a3" Nov 01 21:01:35.003457 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.003348 4477 scope.go:122] "RemoveContainer" containerID="d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171" Nov 01 21:01:35.003581 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.003547 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7295-8eb8ff" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" containerName="test-curl-7295-8eb8ff" containerID="cri-o://cfe2c746852f95a9c802c558119fb3c9f9419c403f2bf4f37d809e890f7997c8" gracePeriod=2 Nov 01 21:01:35.013800 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.013779 4477 scope.go:122] "RemoveContainer" containerID="d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171" Nov 01 21:01:35.014131 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:35.014068 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171\": container with ID starting with d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171 not found: ID does not exist" containerID="d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171" Nov 01 21:01:35.014205 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.014141 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171"} err="failed to get container status \"d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171\": rpc error: code = NotFound desc = could not find container \"d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171\": container with ID starting with d0cc061e0fba8f86cf63d8234d2aaf703d3a5ee38452c2f09414d77e097e1171 not found: ID does not exist" Nov 01 21:01:35.054494 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:35.054449 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-8eb8ff\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:35.105197 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:35.105158 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-8eb8ff\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:35.542702 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.542664 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j"] Nov 01 21:01:35.543820 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.543792 4477 cpu_manager.go:414] "containerMap: removing container" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" containerName="test-curl-7295-8eb8ff" Nov 01 21:01:35.543939 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.543821 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" containerName="test-curl-7295-8eb8ff" Nov 01 21:01:35.544010 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.543997 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" containerName="test-curl-7295-8eb8ff" Nov 01 21:01:35.549831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.549796 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.716099 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:35.715930 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-8eb8ff\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:35.716299 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.716210 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-22ed7b-openshift-default-dockercfg-lsjk5\"" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.718524 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j"] Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.720746 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-socket\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.720789 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-token\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.720896 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-credential-socket\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.720932 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26w9h\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-kube-api-access-26w9h\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.721127 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f81586-ea85-4a06-a08e-66d6f186a24a-istiod-ca-cert\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.721203 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-envoy\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.721263 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-certs\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.721343 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-data\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.749096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.721450 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-podinfo\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.822913 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.822882 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f81586-ea85-4a06-a08e-66d6f186a24a-istiod-ca-cert\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823114 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.822933 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-envoy\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823114 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.822967 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-certs\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823259 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823117 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-data\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823259 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823225 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-podinfo\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823362 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823342 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-socket\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823420 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823371 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-token\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823420 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823390 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-certs\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823668 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823645 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-data\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823787 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823762 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f81586-ea85-4a06-a08e-66d6f186a24a-istiod-ca-cert\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823862 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823782 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-credential-socket\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823862 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823825 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26w9h\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-kube-api-access-26w9h\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.823960 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.823885 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-socket\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.824158 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.824136 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-credential-socket\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.826503 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.826472 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-podinfo\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.826876 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.826855 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-envoy\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.880182 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.880053 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-token\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:35.880182 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:35.880155 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26w9h\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-kube-api-access-26w9h\") pod \"e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:36.177495 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:36.177396 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:36.551917 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:36.551815 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-8eb8ff\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:36.563395 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:36.563355 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:01:36.638430 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:01:36.638386 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f81586_ea85_4a06_a08e_66d6f186a24a.slice/crio-8129f09608f5766d7205bf160e4d0c4227132478dfa1020e63c6e89c2ac206f9 WatchSource:0}: Error finding container 8129f09608f5766d7205bf160e4d0c4227132478dfa1020e63c6e89c2ac206f9: Status 404 returned error can't find the container with id 8129f09608f5766d7205bf160e4d0c4227132478dfa1020e63c6e89c2ac206f9 Nov 01 21:01:36.638699 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:36.638675 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j"] Nov 01 21:01:36.641628 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:36.641592 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:01:36.641748 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:36.641660 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:01:36.641748 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:36.641689 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:01:36.964790 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:36.964754 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:01:37.039816 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.039773 4477 generic.go:356] "Generic (PLEG): container finished" podID="bd31b427-10ae-4070-85d2-bec0d2869034" containerID="cfe2c746852f95a9c802c558119fb3c9f9419c403f2bf4f37d809e890f7997c8" exitCode=137 Nov 01 21:01:37.045954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.045910 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" event={"ID":"b8f81586-ea85-4a06-a08e-66d6f186a24a","Type":"ContainerStarted","Data":"ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac"} Nov 01 21:01:37.046179 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.045969 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" event={"ID":"b8f81586-ea85-4a06-a08e-66d6f186a24a","Type":"ContainerStarted","Data":"8129f09608f5766d7205bf160e4d0c4227132478dfa1020e63c6e89c2ac206f9"} Nov 01 21:01:37.182301 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.181381 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:37.185424 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.185384 4477 patch_prober.go:29] interesting pod/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:01:37.185581 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.185446 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 21:01:37.223429 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.222056 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" podStartSLOduration=2.22203187 podStartE2EDuration="2.22203187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:37.211414273 +0000 UTC m=+1850.861465835" watchObservedRunningTime="2026-11-01 21:01:37.22203187 +0000 UTC m=+1850.872083417" Nov 01 21:01:37.296259 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.296228 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:37.360501 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.360463 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd31b427-10ae-4070-85d2-bec0d2869034-kube-api-access-wc4fm\" (UniqueName: \"kubernetes.io/projected/bd31b427-10ae-4070-85d2-bec0d2869034-kube-api-access-wc4fm\") pod \"bd31b427-10ae-4070-85d2-bec0d2869034\" (UID: \"bd31b427-10ae-4070-85d2-bec0d2869034\") " Nov 01 21:01:37.365730 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.365696 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd31b427-10ae-4070-85d2-bec0d2869034-kube-api-access-wc4fm" pod "bd31b427-10ae-4070-85d2-bec0d2869034" (UID: "bd31b427-10ae-4070-85d2-bec0d2869034"). InnerVolumeSpecName "kube-api-access-wc4fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:01:37.462667 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:37.462636 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4fm\" (UniqueName: \"kubernetes.io/projected/bd31b427-10ae-4070-85d2-bec0d2869034-kube-api-access-wc4fm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:38.056310 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:38.056270 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:38.056777 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:38.056319 4477 scope.go:122] "RemoveContainer" containerID="cfe2c746852f95a9c802c558119fb3c9f9419c403f2bf4f37d809e890f7997c8" Nov 01 21:01:38.115769 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:38.115721 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-8eb8ff\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:38.183499 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:38.183471 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:38.234308 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:01:38.234268 4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-8eb8ff\" is forbidden: User \"system:node:ip-10-0-133-242.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-242.us-west-1.compute.internal' and this object" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" pod="opendatahub/test-curl-7295-8eb8ff" Nov 01 21:01:38.505310 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:38.505281 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd31b427-10ae-4070-85d2-bec0d2869034" path="/var/lib/kubelet/pods/bd31b427-10ae-4070-85d2-bec0d2869034/volumes" Nov 01 21:01:39.064730 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:39.064698 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:39.065820 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:39.065798 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:01:55.825787 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:01:55.825750 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:02:02.948131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:02.948091 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4"] Nov 01 21:02:02.948793 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:02.948762 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" podUID="cb4fefff-e5b4-4136-a877-366aba1d0f8e" containerName="maas-api" containerID="cri-o://aa63ebcc363f33ed7c86f81976fd6eccc7649384fdbef3b55882251e377fbf38" gracePeriod=30 Nov 01 21:02:03.303985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.303953 4477 generic.go:356] "Generic (PLEG): container finished" podID="cb4fefff-e5b4-4136-a877-366aba1d0f8e" containerID="aa63ebcc363f33ed7c86f81976fd6eccc7649384fdbef3b55882251e377fbf38" exitCode=0 Nov 01 21:02:03.304123 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.304104 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" event={"ID":"cb4fefff-e5b4-4136-a877-366aba1d0f8e","Type":"ContainerDied","Data":"aa63ebcc363f33ed7c86f81976fd6eccc7649384fdbef3b55882251e377fbf38"} Nov 01 21:02:03.361097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.359365 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 21:02:03.418098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.417560 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-metrics-certs\") pod \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " Nov 01 21:02:03.418098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.417615 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-maas-api-tls\") pod \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " Nov 01 21:02:03.418098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.417902 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb4fefff-e5b4-4136-a877-366aba1d0f8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb4fefff-e5b4-4136-a877-366aba1d0f8e-openshift-service-ca\") pod \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " Nov 01 21:02:03.418098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.417952 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4fefff-e5b4-4136-a877-366aba1d0f8e-kube-api-access-qw4k6\" (UniqueName: \"kubernetes.io/projected/cb4fefff-e5b4-4136-a877-366aba1d0f8e-kube-api-access-qw4k6\") pod \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\" (UID: \"cb4fefff-e5b4-4136-a877-366aba1d0f8e\") " Nov 01 21:02:03.422110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.419702 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4fefff-e5b4-4136-a877-366aba1d0f8e-openshift-service-ca" pod "cb4fefff-e5b4-4136-a877-366aba1d0f8e" (UID: "cb4fefff-e5b4-4136-a877-366aba1d0f8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:02:03.475106 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.465480 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-maas-api-tls" pod "cb4fefff-e5b4-4136-a877-366aba1d0f8e" (UID: "cb4fefff-e5b4-4136-a877-366aba1d0f8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:02:03.475106 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.465658 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4fefff-e5b4-4136-a877-366aba1d0f8e-kube-api-access-qw4k6" pod "cb4fefff-e5b4-4136-a877-366aba1d0f8e" (UID: "cb4fefff-e5b4-4136-a877-366aba1d0f8e"). InnerVolumeSpecName "kube-api-access-qw4k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:03.477419 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.476520 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-metrics-certs" pod "cb4fefff-e5b4-4136-a877-366aba1d0f8e" (UID: "cb4fefff-e5b4-4136-a877-366aba1d0f8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:02:03.549984 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.530264 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:03.549984 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.530308 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb4fefff-e5b4-4136-a877-366aba1d0f8e-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:03.549984 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.530327 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb4fefff-e5b4-4136-a877-366aba1d0f8e-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:03.549984 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:03.530344 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw4k6\" (UniqueName: \"kubernetes.io/projected/cb4fefff-e5b4-4136-a877-366aba1d0f8e-kube-api-access-qw4k6\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:04.312085 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:04.312055 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" Nov 01 21:02:04.312520 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:04.312109 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4" event={"ID":"cb4fefff-e5b4-4136-a877-366aba1d0f8e","Type":"ContainerDied","Data":"50353a6b9c12f7eff3d2e71b2e5d9141cce0d4f3c6afb8d9d7f9740877c2bf31"} Nov 01 21:02:04.312520 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:04.312175 4477 scope.go:122] "RemoveContainer" containerID="aa63ebcc363f33ed7c86f81976fd6eccc7649384fdbef3b55882251e377fbf38" Nov 01 21:02:04.476692 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:04.476660 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4"] Nov 01 21:02:04.526149 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:04.526114 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-mfwv4"] Nov 01 21:02:05.677579 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:05.677541 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9"] Nov 01 21:02:05.730225 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:05.730192 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-9fvx9"] Nov 01 21:02:06.504586 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:06.504550 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286ccd45-4a2a-4ed7-b424-45e1917569fc" path="/var/lib/kubelet/pods/286ccd45-4a2a-4ed7-b424-45e1917569fc/volumes" Nov 01 21:02:06.505066 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:06.505035 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb4fefff-e5b4-4136-a877-366aba1d0f8e" path="/var/lib/kubelet/pods/cb4fefff-e5b4-4136-a877-366aba1d0f8e/volumes" Nov 01 21:02:06.555331 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:02:06.555290 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:02:20.027579 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.027537 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz"] Nov 01 21:02:20.028384 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.028357 4477 cpu_manager.go:414] "containerMap: removing container" podUID="cb4fefff-e5b4-4136-a877-366aba1d0f8e" containerName="maas-api" Nov 01 21:02:20.028384 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.028380 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb4fefff-e5b4-4136-a877-366aba1d0f8e" containerName="maas-api" Nov 01 21:02:20.028609 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.028595 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb4fefff-e5b4-4136-a877-366aba1d0f8e" containerName="maas-api" Nov 01 21:02:20.032387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.032364 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.089145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.089111 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz"] Nov 01 21:02:20.132438 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.132400 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d08fd0-serving-cert\"" Nov 01 21:02:20.169729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.169669 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80027a69-7918-4633-b561-d080b6d10131-openshift-service-ca\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.169978 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.169883 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-maas-api-tls\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.170119 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.170096 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zmm\" (UniqueName: \"kubernetes.io/projected/80027a69-7918-4633-b561-d080b6d10131-kube-api-access-q6zmm\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.170531 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.170143 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-metrics-certs\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.272060 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.272020 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80027a69-7918-4633-b561-d080b6d10131-openshift-service-ca\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.272314 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.272159 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-maas-api-tls\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.272314 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.272278 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zmm\" (UniqueName: \"kubernetes.io/projected/80027a69-7918-4633-b561-d080b6d10131-kube-api-access-q6zmm\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.272441 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.272326 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-metrics-certs\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.272937 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.272909 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80027a69-7918-4633-b561-d080b6d10131-openshift-service-ca\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.276222 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.276192 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-metrics-certs\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.276375 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.276192 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-maas-api-tls\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.331718 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.331690 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zmm\" (UniqueName: \"kubernetes.io/projected/80027a69-7918-4633-b561-d080b6d10131-kube-api-access-q6zmm\") pod \"maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.350378 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.350343 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:20.736367 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.736341 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz"] Nov 01 21:02:20.737414 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:02:20.737386 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80027a69_7918_4633_b561_d080b6d10131.slice/crio-8060b6926afe1b403ebebc83e3e8d337916711e001d7f99d2ae65b51a7bd9780 WatchSource:0}: Error finding container 8060b6926afe1b403ebebc83e3e8d337916711e001d7f99d2ae65b51a7bd9780: Status 404 returned error can't find the container with id 8060b6926afe1b403ebebc83e3e8d337916711e001d7f99d2ae65b51a7bd9780 Nov 01 21:02:20.738881 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:20.738864 4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:02:21.463995 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:21.463961 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" event={"ID":"80027a69-7918-4633-b561-d080b6d10131","Type":"ContainerStarted","Data":"8060b6926afe1b403ebebc83e3e8d337916711e001d7f99d2ae65b51a7bd9780"} Nov 01 21:02:22.476682 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:22.476643 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" event={"ID":"80027a69-7918-4633-b561-d080b6d10131","Type":"ContainerStarted","Data":"c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21"} Nov 01 21:02:22.477291 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:22.477265 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:22.594457 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:22.594411 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" podStartSLOduration=2.6748236 podStartE2EDuration="3.594395762s" totalImagesPullingTime="919.572162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:22.590214982 +0000 UTC m=+1896.240266538" watchObservedRunningTime="2026-11-01 21:02:22.594395762 +0000 UTC m=+1896.244447306" Nov 01 21:02:28.490906 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:28.490878 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:02:29.951212 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:29.951176 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk"] Nov 01 21:02:29.955641 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:29.955617 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.013541 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.013506 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk"] Nov 01 21:02:30.022772 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.022729 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-metrics-certs\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.022964 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.022831 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88x42\" (UniqueName: \"kubernetes.io/projected/29c45515-da77-4ab0-971a-eb97e86c0f4a-kube-api-access-88x42\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.022964 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.022914 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.023125 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.023107 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c45515-da77-4ab0-971a-eb97e86c0f4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.056605 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.056566 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dd840a-serving-cert\"" Nov 01 21:02:30.124273 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.124235 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-metrics-certs\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.124447 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.124292 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88x42\" (UniqueName: \"kubernetes.io/projected/29c45515-da77-4ab0-971a-eb97e86c0f4a-kube-api-access-88x42\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.124447 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.124369 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.124573 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:02:30.124516 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-serving-cert: secret "maas-api-e2e-api-a-dd840a-serving-cert" not found Nov 01 21:02:30.124573 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.124543 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c45515-da77-4ab0-971a-eb97e86c0f4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.124687 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:02:30.124602 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls podName:29c45515-da77-4ab0-971a-eb97e86c0f4a nodeName:}" failed. No retries permitted until 2026-11-01 21:02:30.624565189 +0000 UTC m=+1904.274616727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls") pod "maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" (UID: "29c45515-da77-4ab0-971a-eb97e86c0f4a") : secret "maas-api-e2e-api-a-dd840a-serving-cert" not found Nov 01 21:02:30.125261 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.125238 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c45515-da77-4ab0-971a-eb97e86c0f4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.126836 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.126814 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-metrics-certs\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.181770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.181731 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88x42\" (UniqueName: \"kubernetes.io/projected/29c45515-da77-4ab0-971a-eb97e86c0f4a-kube-api-access-88x42\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.631819 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.631783 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.634430 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.634401 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls\") pod \"maas-api-e2e-api-a-dd840a-574749fb46-w7nhk\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:30.870363 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:30.870322 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:31.256277 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:31.256245 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk"] Nov 01 21:02:31.257296 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:02:31.257262 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c45515_da77_4ab0_971a_eb97e86c0f4a.slice/crio-184d92de0fac5c15ee0c527dcf5b24784884fee259002409dccc4ea9b0a6c941 WatchSource:0}: Error finding container 184d92de0fac5c15ee0c527dcf5b24784884fee259002409dccc4ea9b0a6c941: Status 404 returned error can't find the container with id 184d92de0fac5c15ee0c527dcf5b24784884fee259002409dccc4ea9b0a6c941 Nov 01 21:02:31.552847 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:31.552762 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" event={"ID":"29c45515-da77-4ab0-971a-eb97e86c0f4a","Type":"ContainerStarted","Data":"184d92de0fac5c15ee0c527dcf5b24784884fee259002409dccc4ea9b0a6c941"} Nov 01 21:02:32.562164 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:32.562138 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" event={"ID":"29c45515-da77-4ab0-971a-eb97e86c0f4a","Type":"ContainerStarted","Data":"781d27788bd9807209b9da910a7e3ed76496027ef1557c40f67dbe2a807ee246"} Nov 01 21:02:32.562655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:32.562216 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:32.677910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:32.677857 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" podStartSLOduration=2.831520042 podStartE2EDuration="3.677842188s" totalImagesPullingTime="846.322146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:32.675301552 +0000 UTC m=+1906.325353144" watchObservedRunningTime="2026-11-01 21:02:32.677842188 +0000 UTC m=+1906.327893759" Nov 01 21:02:36.554176 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:02:36.554133 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:02:38.574051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:38.574024 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:02:45.525092 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:45.525031 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5"] Nov 01 21:02:45.525580 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:45.525415 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" containerID="cri-o://5bec88da5b9177e025ebd399ceed60c0148fe35dc1eb57ea0c57271331289be8" gracePeriod=30 Nov 01 21:02:50.734536 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.734486 4477 generic.go:356] "Generic (PLEG): container finished" podID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerID="5bec88da5b9177e025ebd399ceed60c0148fe35dc1eb57ea0c57271331289be8" exitCode=0 Nov 01 21:02:50.735056 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.735022 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" event={"ID":"e06ec19e-e8fd-4b6e-879e-e5e3092a0315","Type":"ContainerDied","Data":"5bec88da5b9177e025ebd399ceed60c0148fe35dc1eb57ea0c57271331289be8"} Nov 01 21:02:50.816411 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.816386 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 21:02:50.967542 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967503 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-data\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.967542 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967548 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istiod-ca-cert\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.967785 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967616 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-kube-api-access-jwbhz\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-kube-api-access-jwbhz\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.967785 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967680 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-credential-socket\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.967882 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967792 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-data" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:02:50.967882 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967807 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-envoy\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.967882 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967841 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-socket\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.967882 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967865 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-podinfo\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.968174 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.967951 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-certs\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.968174 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968033 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-credential-socket" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:02:50.968174 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968042 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-socket" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:02:50.968174 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968098 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istiod-ca-cert" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:02:50.968174 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968114 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-token\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-token\") pod \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\" (UID: \"e06ec19e-e8fd-4b6e-879e-e5e3092a0315\") " Nov 01 21:02:50.968473 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968216 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-certs" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:02:50.968886 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968856 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.969015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968889 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.969015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968905 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.969015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968919 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.969015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.968933 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.970323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.970295 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-envoy" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:02:50.970414 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.970314 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-kube-api-access-jwbhz" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "kube-api-access-jwbhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:50.970414 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.970374 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-podinfo" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:02:50.970881 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:50.970865 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-token" pod "e06ec19e-e8fd-4b6e-879e-e5e3092a0315" (UID: "e06ec19e-e8fd-4b6e-879e-e5e3092a0315"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:51.070759 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:51.070725 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:51.070759 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:51.070754 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwbhz\" (UniqueName: \"kubernetes.io/projected/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-kube-api-access-jwbhz\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:51.070759 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:51.070766 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:51.071004 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:51.070775 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ec19e-e8fd-4b6e-879e-e5e3092a0315-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:51.745457 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:51.745426 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" Nov 01 21:02:51.745936 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:51.745465 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5" event={"ID":"e06ec19e-e8fd-4b6e-879e-e5e3092a0315","Type":"ContainerDied","Data":"7b9dbdeda0f1cd8283b336f9dee9aec7ef21e857c58c3a9552f9b5c28682ab8a"} Nov 01 21:02:51.745936 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:51.745543 4477 scope.go:122] "RemoveContainer" containerID="5bec88da5b9177e025ebd399ceed60c0148fe35dc1eb57ea0c57271331289be8" Nov 01 21:02:51.963186 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:51.963142 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5"] Nov 01 21:02:52.010699 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:52.010624 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-ttwc5"] Nov 01 21:02:52.504890 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:02:52.504855 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" path="/var/lib/kubelet/pods/e06ec19e-e8fd-4b6e-879e-e5e3092a0315/volumes" Nov 01 21:03:00.214988 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:00.214905 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr"] Nov 01 21:03:00.215613 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:00.215132 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" containerID="cri-o://edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d" gracePeriod=30 Nov 01 21:03:00.552782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:00.552685 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p"] Nov 01 21:03:00.552963 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:00.552931 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" containerID="cri-o://baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652" gracePeriod=30 Nov 01 21:03:01.640817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:01.640785 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm"] Nov 01 21:03:01.641325 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:01.640997 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" containerID="cri-o://da04a22f34b40823d2c7aeb31abacfb53ad56d708463a6abf76a242d470d5af4" gracePeriod=30 Nov 01 21:03:01.892384 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:01.892306 4477 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:03:01.892530 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:01.892402 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:03:02.852390 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:02.852348 4477 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:03:02.852785 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:02.852418 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:03:04.497570 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:04.497524 4477 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:03:04.498102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:04.497594 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:03:05.487437 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.487369 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 21:03:05.565911 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.565880 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-data\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.565965 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c772c60-c4a9-4ed9-9215-f7996d644ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c772c60-c4a9-4ed9-9215-f7996d644ede-istiod-ca-cert\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566005 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-podinfo\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566035 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-envoy\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566150 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-socket\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566173 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-data" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566308 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-kube-api-access-kfkfk\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-kube-api-access-kfkfk\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566365 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-token\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-token\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566438 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-certs\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566470 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-credential-socket\") pod \"9c772c60-c4a9-4ed9-9215-f7996d644ede\" (UID: \"9c772c60-c4a9-4ed9-9215-f7996d644ede\") " Nov 01 21:03:05.566524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.566493 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c772c60-c4a9-4ed9-9215-f7996d644ede-istiod-ca-cert" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:05.567198 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.567153 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-socket" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.567642 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.567617 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-credential-socket" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.568534 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.568496 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-certs" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.568976 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.568889 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.568976 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.568916 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c772c60-c4a9-4ed9-9215-f7996d644ede-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.568976 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.568931 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.568976 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.568945 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.568976 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.568958 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.569280 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.569048 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-envoy" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.569696 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.569490 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-podinfo" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:03:05.569696 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.569534 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-token" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:05.570331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.570311 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-kube-api-access-kfkfk" pod "9c772c60-c4a9-4ed9-9215-f7996d644ede" (UID: "9c772c60-c4a9-4ed9-9215-f7996d644ede"). InnerVolumeSpecName "kube-api-access-kfkfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:05.670970 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.670933 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.670970 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.670971 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.671193 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.670986 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfkfk\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-kube-api-access-kfkfk\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.671193 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.671000 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c772c60-c4a9-4ed9-9215-f7996d644ede-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.832255 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.832218 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 21:03:05.847571 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.847541 4477 generic.go:356] "Generic (PLEG): container finished" podID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerID="edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d" exitCode=0 Nov 01 21:03:05.847756 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.847593 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" Nov 01 21:03:05.847756 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.847618 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" event={"ID":"9c772c60-c4a9-4ed9-9215-f7996d644ede","Type":"ContainerDied","Data":"edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d"} Nov 01 21:03:05.847756 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.847677 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr" event={"ID":"9c772c60-c4a9-4ed9-9215-f7996d644ede","Type":"ContainerDied","Data":"cd3b7442d5ab338eacab18fb67fd4bfc3e896cf264242a71857336a2a762a7f4"} Nov 01 21:03:05.847756 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.847716 4477 scope.go:122] "RemoveContainer" containerID="edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d" Nov 01 21:03:05.849225 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.849208 4477 generic.go:356] "Generic (PLEG): container finished" podID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerID="baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652" exitCode=0 Nov 01 21:03:05.849351 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.849263 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" Nov 01 21:03:05.849351 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.849291 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" event={"ID":"31b6726a-f88f-430f-a5ed-cd29c37edc98","Type":"ContainerDied","Data":"baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652"} Nov 01 21:03:05.849351 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.849344 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p" event={"ID":"31b6726a-f88f-430f-a5ed-cd29c37edc98","Type":"ContainerDied","Data":"42ac04e7ef12de2ede69d285759269471c6c30950ff08248f21c1fbd8c8454fc"} Nov 01 21:03:05.858753 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.858734 4477 scope.go:122] "RemoveContainer" containerID="edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d" Nov 01 21:03:05.859012 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:03:05.858993 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d\": container with ID starting with edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d not found: ID does not exist" containerID="edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d" Nov 01 21:03:05.859118 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.859019 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d"} err="failed to get container status \"edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d\": rpc error: code = NotFound desc = could not find container \"edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d\": container with ID starting with edf416fd17f86f7caf6cd05bc08e1a597ebf9f0d74c496ced0877f855b5aca0d not found: ID does not exist" Nov 01 21:03:05.859118 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.859035 4477 scope.go:122] "RemoveContainer" containerID="baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652" Nov 01 21:03:05.868489 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.868471 4477 scope.go:122] "RemoveContainer" containerID="baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652" Nov 01 21:03:05.868757 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:03:05.868739 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652\": container with ID starting with baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652 not found: ID does not exist" containerID="baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652" Nov 01 21:03:05.868803 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.868765 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652"} err="failed to get container status \"baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652\": rpc error: code = NotFound desc = could not find container \"baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652\": container with ID starting with baf49212ff66462f4069b167abca14365173e32d17311c986bb81430f75fa652 not found: ID does not exist" Nov 01 21:03:05.974936 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.974907 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-credential-socket\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.974942 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-podinfo\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.974984 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-token\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-token\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975033 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-certs\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975179 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-kube-api-access-zwbsf\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-kube-api-access-zwbsf\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975207 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-data\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975242 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-credential-socket" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.975318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975304 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b6726a-f88f-430f-a5ed-cd29c37edc98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6726a-f88f-430f-a5ed-cd29c37edc98-istiod-ca-cert\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975546 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975404 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-socket\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975546 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975434 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-envoy\") pod \"31b6726a-f88f-430f-a5ed-cd29c37edc98\" (UID: \"31b6726a-f88f-430f-a5ed-cd29c37edc98\") " Nov 01 21:03:05.975546 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975451 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-certs" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.975717 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975655 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-socket" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.975803 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975776 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-data" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.975868 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.975813 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b6726a-f88f-430f-a5ed-cd29c37edc98-istiod-ca-cert" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:05.976163 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.976138 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.976258 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.976167 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.976258 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.976182 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.976258 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.976194 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.976258 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.976206 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6726a-f88f-430f-a5ed-cd29c37edc98-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:05.978109 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.978049 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-podinfo" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:03:05.978211 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.978123 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-token" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:05.978211 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.978187 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-envoy" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:05.978289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:05.978216 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-kube-api-access-zwbsf" pod "31b6726a-f88f-430f-a5ed-cd29c37edc98" (UID: "31b6726a-f88f-430f-a5ed-cd29c37edc98"). InnerVolumeSpecName "kube-api-access-zwbsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:06.077831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.077758 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwbsf\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-kube-api-access-zwbsf\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.077831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.077786 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.077831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.077796 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.077831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.077806 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b6726a-f88f-430f-a5ed-cd29c37edc98-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.113246 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.113211 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr"] Nov 01 21:03:06.117325 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.117293 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-sbntr"] Nov 01 21:03:06.271984 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.271943 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p"] Nov 01 21:03:06.275509 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.275485 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-xqm2p"] Nov 01 21:03:06.506322 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.506234 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" path="/var/lib/kubelet/pods/31b6726a-f88f-430f-a5ed-cd29c37edc98/volumes" Nov 01 21:03:06.506983 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.506966 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" path="/var/lib/kubelet/pods/9c772c60-c4a9-4ed9-9215-f7996d644ede/volumes" Nov 01 21:03:06.559497 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:03:06.559441 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:03:06.869745 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.869707 4477 generic.go:356] "Generic (PLEG): container finished" podID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerID="da04a22f34b40823d2c7aeb31abacfb53ad56d708463a6abf76a242d470d5af4" exitCode=0 Nov 01 21:03:06.870250 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.869823 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" event={"ID":"9e67489e-a66a-454a-acc2-37d86b4cfa84","Type":"ContainerDied","Data":"da04a22f34b40823d2c7aeb31abacfb53ad56d708463a6abf76a242d470d5af4"} Nov 01 21:03:06.915271 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.915248 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 21:03:06.988467 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988434 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-envoy\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.988643 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988482 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-podinfo\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.988643 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988609 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-socket\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.988770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988733 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-credential-socket\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.988822 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988812 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-certs\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.988874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988848 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-data\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.988874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988853 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-socket" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:06.988949 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988908 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-token\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-token\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.988999 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988950 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e67489e-a66a-454a-acc2-37d86b4cfa84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67489e-a66a-454a-acc2-37d86b4cfa84-istiod-ca-cert\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.988999 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988986 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-credential-socket" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:06.989122 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.988991 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-kube-api-access-bq92g\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-kube-api-access-bq92g\") pod \"9e67489e-a66a-454a-acc2-37d86b4cfa84\" (UID: \"9e67489e-a66a-454a-acc2-37d86b4cfa84\") " Nov 01 21:03:06.989122 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.989036 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-certs" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:06.989243 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.989140 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-data" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:06.989453 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.989375 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e67489e-a66a-454a-acc2-37d86b4cfa84-istiod-ca-cert" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:06.989937 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.989908 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.990061 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.989941 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.990061 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.989956 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67489e-a66a-454a-acc2-37d86b4cfa84-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.990061 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.989969 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.990061 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.989983 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:06.990995 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.990974 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-podinfo" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:03:06.991240 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.991218 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-envoy" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:06.991545 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.991527 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-kube-api-access-bq92g" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "kube-api-access-bq92g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:06.991600 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:06.991571 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-token" pod "9e67489e-a66a-454a-acc2-37d86b4cfa84" (UID: "9e67489e-a66a-454a-acc2-37d86b4cfa84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:07.091601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:07.091517 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:07.091601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:07.091546 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq92g\" (UniqueName: \"kubernetes.io/projected/9e67489e-a66a-454a-acc2-37d86b4cfa84-kube-api-access-bq92g\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:07.091601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:07.091561 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:07.091601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:07.091573 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67489e-a66a-454a-acc2-37d86b4cfa84-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:07.880027 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:07.879999 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" Nov 01 21:03:07.880547 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:07.880033 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm" event={"ID":"9e67489e-a66a-454a-acc2-37d86b4cfa84","Type":"ContainerDied","Data":"255feb1bad91daf7154990ec44eadedb82dd137610f1f3cde9666232c7b4bd26"} Nov 01 21:03:07.880547 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:07.880126 4477 scope.go:122] "RemoveContainer" containerID="da04a22f34b40823d2c7aeb31abacfb53ad56d708463a6abf76a242d470d5af4" Nov 01 21:03:08.104443 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:08.104410 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm"] Nov 01 21:03:08.154434 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:08.154355 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-vcdwm"] Nov 01 21:03:08.506190 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:08.506113 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" path="/var/lib/kubelet/pods/9e67489e-a66a-454a-acc2-37d86b4cfa84/volumes" Nov 01 21:03:13.579756 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.579724 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b"] Nov 01 21:03:13.580375 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580357 4477 cpu_manager.go:414] "containerMap: removing container" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" Nov 01 21:03:13.580423 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580378 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" Nov 01 21:03:13.580423 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580395 4477 cpu_manager.go:414] "containerMap: removing container" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" Nov 01 21:03:13.580423 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580401 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" Nov 01 21:03:13.580423 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580421 4477 cpu_manager.go:414] "containerMap: removing container" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" Nov 01 21:03:13.580550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580426 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" Nov 01 21:03:13.580550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580439 4477 cpu_manager.go:414] "containerMap: removing container" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" Nov 01 21:03:13.580550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580446 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" Nov 01 21:03:13.580550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580550 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c772c60-c4a9-4ed9-9215-f7996d644ede" containerName="istio-proxy" Nov 01 21:03:13.580705 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580560 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b6726a-f88f-430f-a5ed-cd29c37edc98" containerName="istio-proxy" Nov 01 21:03:13.580705 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580570 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06ec19e-e8fd-4b6e-879e-e5e3092a0315" containerName="istio-proxy" Nov 01 21:03:13.580705 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.580576 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e67489e-a66a-454a-acc2-37d86b4cfa84" containerName="istio-proxy" Nov 01 21:03:13.584705 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.584687 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.683589 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.683560 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-metrics-certs\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.683841 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.683811 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.684054 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.684030 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvsv\" (UniqueName: \"kubernetes.io/projected/26f1dd72-fa2d-4a16-bbde-2199c9422788-kube-api-access-2rvsv\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.684271 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.684246 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f1dd72-fa2d-4a16-bbde-2199c9422788-openshift-service-ca\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.686369 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.686351 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-081c8e-serving-cert\"" Nov 01 21:03:13.690180 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.690161 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b"] Nov 01 21:03:13.785990 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.785943 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.786202 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.786105 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvsv\" (UniqueName: \"kubernetes.io/projected/26f1dd72-fa2d-4a16-bbde-2199c9422788-kube-api-access-2rvsv\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.786202 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:03:13.786138 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-serving-cert: secret "maas-api-e2e-inf-a-081c8e-serving-cert" not found Nov 01 21:03:13.786202 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.786177 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f1dd72-fa2d-4a16-bbde-2199c9422788-openshift-service-ca\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.786376 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:03:13.786240 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls podName:26f1dd72-fa2d-4a16-bbde-2199c9422788 nodeName:}" failed. No retries permitted until 2026-11-01 21:03:14.286216312 +0000 UTC m=+1947.936267871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls") pod "maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" (UID: "26f1dd72-fa2d-4a16-bbde-2199c9422788") : secret "maas-api-e2e-inf-a-081c8e-serving-cert" not found Nov 01 21:03:13.786376 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.786340 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-metrics-certs\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.787008 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.786980 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f1dd72-fa2d-4a16-bbde-2199c9422788-openshift-service-ca\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.789217 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.789190 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-metrics-certs\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:13.843852 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:13.843782 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvsv\" (UniqueName: \"kubernetes.io/projected/26f1dd72-fa2d-4a16-bbde-2199c9422788-kube-api-access-2rvsv\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:14.294518 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:14.294437 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:14.297184 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:14.297154 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls\") pod \"maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:14.498874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:14.498839 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:14.887924 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:14.887891 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b"] Nov 01 21:03:14.888293 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:03:14.888124 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f1dd72_fa2d_4a16_bbde_2199c9422788.slice/crio-c74f418abd2a83e88680e73ee3cd20818319aaaa212a771090b6cccbab605ff2 WatchSource:0}: Error finding container c74f418abd2a83e88680e73ee3cd20818319aaaa212a771090b6cccbab605ff2: Status 404 returned error can't find the container with id c74f418abd2a83e88680e73ee3cd20818319aaaa212a771090b6cccbab605ff2 Nov 01 21:03:14.938375 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:14.938328 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" event={"ID":"26f1dd72-fa2d-4a16-bbde-2199c9422788","Type":"ContainerStarted","Data":"c74f418abd2a83e88680e73ee3cd20818319aaaa212a771090b6cccbab605ff2"} Nov 01 21:03:15.952608 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:15.952571 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" event={"ID":"26f1dd72-fa2d-4a16-bbde-2199c9422788","Type":"ContainerStarted","Data":"6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9"} Nov 01 21:03:15.953145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:15.952701 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:16.070510 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:16.070451 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" podStartSLOduration=2.237068784 podStartE2EDuration="3.070435251s" totalImagesPullingTime="833.366467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:16.065861902 +0000 UTC m=+1949.715913449" watchObservedRunningTime="2026-11-01 21:03:16.070435251 +0000 UTC m=+1949.720486796" Nov 01 21:03:21.966987 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:21.966955 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:03:22.952582 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:22.952545 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:03:35.417781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.417742 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2"] Nov 01 21:03:35.418241 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.417959 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" podUID="d6c53d18-8bfa-4fd1-8622-41c885aee1fd" containerName="discovery" containerID="cri-o://ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8" gracePeriod=30 Nov 01 21:03:35.711996 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.711970 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 21:03:35.896049 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.896015 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-ca-configmap\") pod \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " Nov 01 21:03:35.896049 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.896068 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-cacerts\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-cacerts\") pod \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " Nov 01 21:03:35.896347 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.896108 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-kube-api-access-72h9c\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-kube-api-access-72h9c\") pod \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " Nov 01 21:03:35.896347 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.896184 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-kubeconfig\") pod \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " Nov 01 21:03:35.896347 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.896221 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-local-certs\") pod \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " Nov 01 21:03:35.896347 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.896302 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-token\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-token\") pod \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " Nov 01 21:03:35.896566 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.896450 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-dns-cert\") pod \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\" (UID: \"d6c53d18-8bfa-4fd1-8622-41c885aee1fd\") " Nov 01 21:03:35.896566 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.896498 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-ca-configmap" pod "d6c53d18-8bfa-4fd1-8622-41c885aee1fd" (UID: "d6c53d18-8bfa-4fd1-8622-41c885aee1fd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:35.897247 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.897224 4477 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-ca-configmap\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:35.899316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.899284 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-token" pod "d6c53d18-8bfa-4fd1-8622-41c885aee1fd" (UID: "d6c53d18-8bfa-4fd1-8622-41c885aee1fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:35.899546 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.899521 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-local-certs" pod "d6c53d18-8bfa-4fd1-8622-41c885aee1fd" (UID: "d6c53d18-8bfa-4fd1-8622-41c885aee1fd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:35.899626 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.899556 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-kubeconfig" pod "d6c53d18-8bfa-4fd1-8622-41c885aee1fd" (UID: "d6c53d18-8bfa-4fd1-8622-41c885aee1fd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:03:35.899626 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.899583 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-cacerts" pod "d6c53d18-8bfa-4fd1-8622-41c885aee1fd" (UID: "d6c53d18-8bfa-4fd1-8622-41c885aee1fd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:03:35.899868 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.899848 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-kube-api-access-72h9c" pod "d6c53d18-8bfa-4fd1-8622-41c885aee1fd" (UID: "d6c53d18-8bfa-4fd1-8622-41c885aee1fd"). InnerVolumeSpecName "kube-api-access-72h9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:35.901055 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.901029 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-dns-cert" pod "d6c53d18-8bfa-4fd1-8622-41c885aee1fd" (UID: "d6c53d18-8bfa-4fd1-8622-41c885aee1fd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:03:35.998848 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.998755 4477 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-cacerts\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:35.998848 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.998797 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72h9c\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-kube-api-access-72h9c\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:35.998848 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.998810 4477 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-kubeconfig\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:35.998848 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.998820 4477 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-local-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:35.998848 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.998829 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:35.998848 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:35.998838 4477 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6c53d18-8bfa-4fd1-8622-41c885aee1fd-istio-csr-dns-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:36.111702 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.111660 4477 generic.go:356] "Generic (PLEG): container finished" podID="d6c53d18-8bfa-4fd1-8622-41c885aee1fd" containerID="ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8" exitCode=0 Nov 01 21:03:36.111908 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.111715 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" Nov 01 21:03:36.111908 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.111741 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" event={"ID":"d6c53d18-8bfa-4fd1-8622-41c885aee1fd","Type":"ContainerDied","Data":"ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8"} Nov 01 21:03:36.111908 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.111790 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2" event={"ID":"d6c53d18-8bfa-4fd1-8622-41c885aee1fd","Type":"ContainerDied","Data":"0101d0e917ea9a5e046146e55c1e5d43a8f30084f8170dbe3223fe67ee2f06bb"} Nov 01 21:03:36.111908 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.111815 4477 scope.go:122] "RemoveContainer" containerID="ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8" Nov 01 21:03:36.122903 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.122885 4477 scope.go:122] "RemoveContainer" containerID="ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8" Nov 01 21:03:36.123209 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:03:36.123188 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8\": container with ID starting with ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8 not found: ID does not exist" containerID="ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8" Nov 01 21:03:36.123257 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.123220 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8"} err="failed to get container status \"ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8\": rpc error: code = NotFound desc = could not find container \"ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8\": container with ID starting with ba5ecd348c95ff9bbab5a360e901d0b1df8bb1da4a6d22d10cb79f6765cfd0e8 not found: ID does not exist" Nov 01 21:03:36.276162 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.276049 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2"] Nov 01 21:03:36.325810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.325784 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qg2c2"] Nov 01 21:03:36.505773 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:03:36.505743 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c53d18-8bfa-4fd1-8622-41c885aee1fd" path="/var/lib/kubelet/pods/d6c53d18-8bfa-4fd1-8622-41c885aee1fd/volumes" Nov 01 21:03:36.561334 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:03:36.561252 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:04:00.340043 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.340008 4477 generic.go:356] "Generic (PLEG): container finished" podID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerID="ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac" exitCode=0 Nov 01 21:04:00.340500 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.340120 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" event={"ID":"b8f81586-ea85-4a06-a08e-66d6f186a24a","Type":"ContainerDied","Data":"ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac"} Nov 01 21:04:00.340588 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.340573 4477 scope.go:122] "RemoveContainer" containerID="ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac" Nov 01 21:04:00.342460 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.342429 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:04:00.342561 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.342509 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:04:00.342561 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.342544 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:04:00.557678 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.557639 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l"] Nov 01 21:04:00.557923 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.557893 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" containerID="cri-o://8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8" gracePeriod=30 Nov 01 21:04:00.562230 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.562204 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k"] Nov 01 21:04:00.562436 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.562412 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" containerID="cri-o://77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b" gracePeriod=30 Nov 01 21:04:00.565416 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.565395 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j"] Nov 01 21:04:00.565604 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.565582 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" containerID="cri-o://53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c" gracePeriod=30 Nov 01 21:04:00.567953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.567900 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk"] Nov 01 21:04:00.568115 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:00.568094 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" containerID="cri-o://bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c" gracePeriod=30 Nov 01 21:04:01.350401 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:01.350368 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" event={"ID":"b8f81586-ea85-4a06-a08e-66d6f186a24a","Type":"ContainerStarted","Data":"5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6"} Nov 01 21:04:02.177963 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:02.177931 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:04:02.178282 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:02.178257 4477 patch_prober.go:29] interesting pod/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:04:02.178353 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:02.178328 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 21:04:03.177958 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:03.177924 4477 patch_prober.go:29] interesting pod/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:04:03.178480 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:03.177994 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 21:04:04.178389 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:04.178353 4477 patch_prober.go:29] interesting pod/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:04:04.178801 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:04.178405 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 21:04:05.178628 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:05.178582 4477 patch_prober.go:29] interesting pod/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:04:05.179059 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:05.178650 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 21:04:05.291151 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:05.291116 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pfnvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:04:05.291314 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:05.291189 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:04:05.443417 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:05.443321 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qzw6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:04:05.443743 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:05.443693 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:04:05.962969 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:05.960985 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 21:04:06.011181 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.011153 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 21:04:06.027103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.021334 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 21:04:06.099475 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.099442 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-credential-socket\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.099629 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.099495 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-data\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.099629 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.099539 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-kube-api-access-6hjjz\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-kube-api-access-6hjjz\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.099629 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.099597 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-socket\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.099981 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.099958 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-certs\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.100271 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-credential-socket" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.100725 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-socket" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.100759 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-data" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.100834 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-socket\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.100876 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-credential-socket\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.100989 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-certs" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101134 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istiod-ca-cert\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101167 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-envoy\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101231 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-credential-socket" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101235 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-kube-api-access-wj9zk\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-kube-api-access-wj9zk\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101313 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-podinfo\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101355 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-podinfo\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101388 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-podinfo\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101438 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istiod-ca-cert\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101464 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-token\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101484 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istiod-ca-cert" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101497 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-credential-socket\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101532 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-kube-api-access-s88f4\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-kube-api-access-s88f4\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.102810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.101563 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-data\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103088 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-data\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103138 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-token\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-token\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103176 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-certs\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103221 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-token\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-token\") pod \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\" (UID: \"9918869f-25aa-4fbb-ae9f-3881dfd069ca\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103254 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istiod-ca-cert\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103307 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-certs\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103340 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-envoy\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103393 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-socket\") pod \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\" (UID: \"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba\") " Nov 01 21:04:06.103766 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103424 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-envoy\") pod \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\" (UID: \"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d\") " Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104276 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104295 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104308 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104322 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104349 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104365 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.102972 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-credential-socket" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.102991 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-socket" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.103024 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-data" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104337 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-podinfo" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:04:06.104622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104611 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istiod-ca-cert" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:06.105308 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.104851 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-data" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.105308 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.105252 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-certs" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.105655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.105435 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-certs" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.105746 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.105709 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istiod-ca-cert" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:06.106939 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.106913 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-socket" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.108786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.108745 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-kube-api-access-6hjjz" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "kube-api-access-6hjjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:06.110657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.110633 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-podinfo" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:04:06.111032 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.110991 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-token" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:06.111196 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.111111 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-kube-api-access-wj9zk" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "kube-api-access-wj9zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:06.111292 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.111270 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-token" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:06.111454 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.111431 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-envoy" pod "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" (UID: "b1c3b7d1-0ee5-4e04-a1cd-0f071987095d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.111545 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.111526 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-podinfo" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:04:06.111929 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.111907 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-envoy" pod "9918869f-25aa-4fbb-ae9f-3881dfd069ca" (UID: "9918869f-25aa-4fbb-ae9f-3881dfd069ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.112312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.112289 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-kube-api-access-s88f4" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "kube-api-access-s88f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:06.112512 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.112494 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-envoy" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.150133 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.146668 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-token" pod "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" (UID: "d9bfedee-7a85-49a9-9ae4-f607aca1f8ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:06.178166 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.178135 4477 patch_prober.go:29] interesting pod/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:04:06.178341 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.178184 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205717 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205745 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205755 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hjjz\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-kube-api-access-6hjjz\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205767 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205775 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205784 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj9zk\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-kube-api-access-wj9zk\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205793 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205802 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205810 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205818 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205826 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205834 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205843 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88f4\" (UniqueName: \"kubernetes.io/projected/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-kube-api-access-s88f4\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205851 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205858 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205866 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205874 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918869f-25aa-4fbb-ae9f-3881dfd069ca-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205883 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918869f-25aa-4fbb-ae9f-3881dfd069ca-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205891 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205899 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.208003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.205908 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.375445 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.374836 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 21:04:06.387996 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.387964 4477 generic.go:356] "Generic (PLEG): container finished" podID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerID="bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c" exitCode=0 Nov 01 21:04:06.388175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.388016 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" Nov 01 21:04:06.388175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.388046 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" event={"ID":"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba","Type":"ContainerDied","Data":"bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c"} Nov 01 21:04:06.388175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.388132 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk" event={"ID":"d9bfedee-7a85-49a9-9ae4-f607aca1f8ba","Type":"ContainerDied","Data":"054ed24b6005a2a38c769e964e73eef0740a483460b316cc411508285b96b9d4"} Nov 01 21:04:06.388175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.388162 4477 scope.go:122] "RemoveContainer" containerID="bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c" Nov 01 21:04:06.389491 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.389456 4477 generic.go:356] "Generic (PLEG): container finished" podID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerID="53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c" exitCode=0 Nov 01 21:04:06.389604 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.389511 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" Nov 01 21:04:06.389684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.389635 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" event={"ID":"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb","Type":"ContainerDied","Data":"53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c"} Nov 01 21:04:06.389788 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.389680 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j" event={"ID":"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb","Type":"ContainerDied","Data":"92a066780555d26283d3960c9bc7435893e49c4715dda8a0de6bbabef4c7d75d"} Nov 01 21:04:06.391401 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.391377 4477 generic.go:356] "Generic (PLEG): container finished" podID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerID="77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b" exitCode=0 Nov 01 21:04:06.391515 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.391433 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" Nov 01 21:04:06.391515 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.391461 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" event={"ID":"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d","Type":"ContainerDied","Data":"77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b"} Nov 01 21:04:06.391627 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.391514 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k" event={"ID":"b1c3b7d1-0ee5-4e04-a1cd-0f071987095d","Type":"ContainerDied","Data":"a78345e65a71bfefef463b910aef350e9f154607b20cca92bf203bedd0fa980c"} Nov 01 21:04:06.393735 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.393104 4477 generic.go:356] "Generic (PLEG): container finished" podID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerID="8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8" exitCode=0 Nov 01 21:04:06.393735 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.393164 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" event={"ID":"9918869f-25aa-4fbb-ae9f-3881dfd069ca","Type":"ContainerDied","Data":"8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8"} Nov 01 21:04:06.393735 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.393202 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" event={"ID":"9918869f-25aa-4fbb-ae9f-3881dfd069ca","Type":"ContainerDied","Data":"7db756c3c22ed79598ba61d8785856ba1e5a9a6afd99d17b8ed9a6a824f99359"} Nov 01 21:04:06.393735 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.393271 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l" Nov 01 21:04:06.408470 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408444 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-credential-socket\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.408565 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408489 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-token\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-token\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.408565 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408530 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istiod-ca-cert\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.408646 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408583 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-kube-api-access-gjj8x\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-kube-api-access-gjj8x\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.408646 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408630 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-podinfo\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.408705 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408662 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-certs\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.408705 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408698 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-envoy\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.408923 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408906 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-socket\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.408982 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.408969 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-data\") pod \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\" (UID: \"8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb\") " Nov 01 21:04:06.410877 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.410360 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-certs" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.410877 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.410581 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-credential-socket" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.410877 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.410630 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istiod-ca-cert" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:06.411220 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.410996 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-socket" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.411381 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.411358 4477 scope.go:122] "RemoveContainer" containerID="bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c" Nov 01 21:04:06.412973 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:04:06.411903 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c\": container with ID starting with bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c not found: ID does not exist" containerID="bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c" Nov 01 21:04:06.412973 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.411938 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c"} err="failed to get container status \"bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c\": rpc error: code = NotFound desc = could not find container \"bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c\": container with ID starting with bd165c304084c4cf6770ac74b5f03ca0437c6598a8ca09830818c9e64b4d875c not found: ID does not exist" Nov 01 21:04:06.412973 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.411961 4477 scope.go:122] "RemoveContainer" containerID="53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c" Nov 01 21:04:06.412973 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.412487 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-data" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.428349 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.426142 4477 scope.go:122] "RemoveContainer" containerID="53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c" Nov 01 21:04:06.428349 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:04:06.426443 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c\": container with ID starting with 53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c not found: ID does not exist" containerID="53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c" Nov 01 21:04:06.428349 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.426473 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c"} err="failed to get container status \"53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c\": rpc error: code = NotFound desc = could not find container \"53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c\": container with ID starting with 53bff1fb5cb0b123f57979d44eecf9b6ca1cdfb52710f6693349e70a1dbeb48c not found: ID does not exist" Nov 01 21:04:06.428349 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.426507 4477 scope.go:122] "RemoveContainer" containerID="77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b" Nov 01 21:04:06.438287 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.438159 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-podinfo" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:04:06.438287 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.438221 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-kube-api-access-gjj8x" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "kube-api-access-gjj8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:06.438287 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.438247 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-token" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:06.438555 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.438304 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-envoy" pod "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" (UID: "8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:06.442750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.442680 4477 scope.go:122] "RemoveContainer" containerID="77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b" Nov 01 21:04:06.443225 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:04:06.443121 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b\": container with ID starting with 77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b not found: ID does not exist" containerID="77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b" Nov 01 21:04:06.443225 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.443177 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b"} err="failed to get container status \"77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b\": rpc error: code = NotFound desc = could not find container \"77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b\": container with ID starting with 77a4cb5387bd650e7e598633617b42fbba6dbaa3e6c6cc0ab77d26f4f4e3ce6b not found: ID does not exist" Nov 01 21:04:06.443464 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.443205 4477 scope.go:122] "RemoveContainer" containerID="8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8" Nov 01 21:04:06.458940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.458395 4477 scope.go:122] "RemoveContainer" containerID="8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8" Nov 01 21:04:06.458940 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:04:06.458724 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8\": container with ID starting with 8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8 not found: ID does not exist" containerID="8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8" Nov 01 21:04:06.458940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.458758 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8"} err="failed to get container status \"8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8\": rpc error: code = NotFound desc = could not find container \"8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8\": container with ID starting with 8eacc7503a326445126c25e51b267534df5b25fcaf172114ab949f887dbc08d8 not found: ID does not exist" Nov 01 21:04:06.515293 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515262 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.515293 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515293 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.515497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515307 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.515497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515319 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.515497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515330 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.515497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515341 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.515497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515356 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.515497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515367 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjj8x\" (UniqueName: \"kubernetes.io/projected/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-kube-api-access-gjj8x\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.515497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.515378 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.560772 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:04:06.560733 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:04:06.797141 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.797031 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk"] Nov 01 21:04:06.803235 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.803200 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pfnvk"] Nov 01 21:04:06.912305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.912270 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k"] Nov 01 21:04:06.915692 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:06.915666 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rfb2k"] Nov 01 21:04:07.024185 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:07.024150 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l"] Nov 01 21:04:07.027413 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:07.027392 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cxh5l"] Nov 01 21:04:07.134598 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:07.134560 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j"] Nov 01 21:04:07.137972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:07.137940 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qzw6j"] Nov 01 21:04:07.182973 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:07.182941 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:04:07.183409 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:07.183387 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:04:07.184170 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:07.184151 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:04:08.504328 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:08.504301 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" path="/var/lib/kubelet/pods/8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb/volumes" Nov 01 21:04:08.504789 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:08.504775 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" path="/var/lib/kubelet/pods/9918869f-25aa-4fbb-ae9f-3881dfd069ca/volumes" Nov 01 21:04:08.505197 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:08.505183 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" path="/var/lib/kubelet/pods/b1c3b7d1-0ee5-4e04-a1cd-0f071987095d/volumes" Nov 01 21:04:08.505890 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:08.505876 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" path="/var/lib/kubelet/pods/d9bfedee-7a85-49a9-9ae4-f607aca1f8ba/volumes" Nov 01 21:04:16.438193 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:16.438150 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:04:19.158159 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.158125 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk"] Nov 01 21:04:19.158686 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.158333 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" podUID="29c45515-da77-4ab0-971a-eb97e86c0f4a" containerName="maas-api" containerID="cri-o://781d27788bd9807209b9da910a7e3ed76496027ef1557c40f67dbe2a807ee246" gracePeriod=30 Nov 01 21:04:19.503443 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.503369 4477 generic.go:356] "Generic (PLEG): container finished" podID="29c45515-da77-4ab0-971a-eb97e86c0f4a" containerID="781d27788bd9807209b9da910a7e3ed76496027ef1557c40f67dbe2a807ee246" exitCode=0 Nov 01 21:04:19.503667 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.503424 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" event={"ID":"29c45515-da77-4ab0-971a-eb97e86c0f4a","Type":"ContainerDied","Data":"781d27788bd9807209b9da910a7e3ed76496027ef1557c40f67dbe2a807ee246"} Nov 01 21:04:19.591341 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.591293 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:04:19.682609 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.682581 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c45515-da77-4ab0-971a-eb97e86c0f4a-kube-api-access-88x42\" (UniqueName: \"kubernetes.io/projected/29c45515-da77-4ab0-971a-eb97e86c0f4a-kube-api-access-88x42\") pod \"29c45515-da77-4ab0-971a-eb97e86c0f4a\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " Nov 01 21:04:19.682883 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.682861 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls\") pod \"29c45515-da77-4ab0-971a-eb97e86c0f4a\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " Nov 01 21:04:19.683137 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.683120 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-metrics-certs\") pod \"29c45515-da77-4ab0-971a-eb97e86c0f4a\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " Nov 01 21:04:19.683560 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.683537 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c45515-da77-4ab0-971a-eb97e86c0f4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c45515-da77-4ab0-971a-eb97e86c0f4a-openshift-service-ca\") pod \"29c45515-da77-4ab0-971a-eb97e86c0f4a\" (UID: \"29c45515-da77-4ab0-971a-eb97e86c0f4a\") " Nov 01 21:04:19.683922 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.683899 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c45515-da77-4ab0-971a-eb97e86c0f4a-openshift-service-ca" pod "29c45515-da77-4ab0-971a-eb97e86c0f4a" (UID: "29c45515-da77-4ab0-971a-eb97e86c0f4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:19.684473 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.684453 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c45515-da77-4ab0-971a-eb97e86c0f4a-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:19.685120 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.685064 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c45515-da77-4ab0-971a-eb97e86c0f4a-kube-api-access-88x42" pod "29c45515-da77-4ab0-971a-eb97e86c0f4a" (UID: "29c45515-da77-4ab0-971a-eb97e86c0f4a"). InnerVolumeSpecName "kube-api-access-88x42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:19.685500 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.685483 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls" pod "29c45515-da77-4ab0-971a-eb97e86c0f4a" (UID: "29c45515-da77-4ab0-971a-eb97e86c0f4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:19.685716 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.685685 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-metrics-certs" pod "29c45515-da77-4ab0-971a-eb97e86c0f4a" (UID: "29c45515-da77-4ab0-971a-eb97e86c0f4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:19.785761 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.785675 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88x42\" (UniqueName: \"kubernetes.io/projected/29c45515-da77-4ab0-971a-eb97e86c0f4a-kube-api-access-88x42\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:19.785761 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.785717 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:19.785761 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:19.785732 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c45515-da77-4ab0-971a-eb97e86c0f4a-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:20.511489 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:20.511441 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" Nov 01 21:04:20.511933 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:20.511479 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk" event={"ID":"29c45515-da77-4ab0-971a-eb97e86c0f4a","Type":"ContainerDied","Data":"184d92de0fac5c15ee0c527dcf5b24784884fee259002409dccc4ea9b0a6c941"} Nov 01 21:04:20.511933 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:20.511546 4477 scope.go:122] "RemoveContainer" containerID="781d27788bd9807209b9da910a7e3ed76496027ef1557c40f67dbe2a807ee246" Nov 01 21:04:20.703306 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:20.703276 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk"] Nov 01 21:04:20.753360 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:20.753317 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd840a-574749fb46-w7nhk"] Nov 01 21:04:22.504256 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:22.504219 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c45515-da77-4ab0-971a-eb97e86c0f4a" path="/var/lib/kubelet/pods/29c45515-da77-4ab0-971a-eb97e86c0f4a/volumes" Nov 01 21:04:29.884634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:29.884548 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm"] Nov 01 21:04:29.885037 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:29.884756 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" podUID="760a62bd-4672-4367-a6df-55b3a23b13c2" containerName="maas-api" containerID="cri-o://9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344" gracePeriod=30 Nov 01 21:04:30.217650 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.217618 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 21:04:30.362922 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.362877 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760a62bd-4672-4367-a6df-55b3a23b13c2-kube-api-access-tqlc2\" (UniqueName: \"kubernetes.io/projected/760a62bd-4672-4367-a6df-55b3a23b13c2-kube-api-access-tqlc2\") pod \"760a62bd-4672-4367-a6df-55b3a23b13c2\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " Nov 01 21:04:30.364939 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.363007 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760a62bd-4672-4367-a6df-55b3a23b13c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760a62bd-4672-4367-a6df-55b3a23b13c2-openshift-service-ca\") pod \"760a62bd-4672-4367-a6df-55b3a23b13c2\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " Nov 01 21:04:30.364939 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.363114 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls\") pod \"760a62bd-4672-4367-a6df-55b3a23b13c2\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " Nov 01 21:04:30.364939 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.363149 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-metrics-certs\") pod \"760a62bd-4672-4367-a6df-55b3a23b13c2\" (UID: \"760a62bd-4672-4367-a6df-55b3a23b13c2\") " Nov 01 21:04:30.373159 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.372975 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760a62bd-4672-4367-a6df-55b3a23b13c2-kube-api-access-tqlc2" pod "760a62bd-4672-4367-a6df-55b3a23b13c2" (UID: "760a62bd-4672-4367-a6df-55b3a23b13c2"). InnerVolumeSpecName "kube-api-access-tqlc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:30.374955 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.374922 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760a62bd-4672-4367-a6df-55b3a23b13c2-openshift-service-ca" pod "760a62bd-4672-4367-a6df-55b3a23b13c2" (UID: "760a62bd-4672-4367-a6df-55b3a23b13c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:30.388247 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.387955 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760a62bd-4672-4367-a6df-55b3a23b13c2-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:30.388247 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.387979 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqlc2\" (UniqueName: \"kubernetes.io/projected/760a62bd-4672-4367-a6df-55b3a23b13c2-kube-api-access-tqlc2\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:30.391429 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.390133 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls" pod "760a62bd-4672-4367-a6df-55b3a23b13c2" (UID: "760a62bd-4672-4367-a6df-55b3a23b13c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:30.395306 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.395259 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-metrics-certs" pod "760a62bd-4672-4367-a6df-55b3a23b13c2" (UID: "760a62bd-4672-4367-a6df-55b3a23b13c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:30.489736 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.489651 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:30.489736 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.489680 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760a62bd-4672-4367-a6df-55b3a23b13c2-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:30.618775 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.618740 4477 generic.go:356] "Generic (PLEG): container finished" podID="760a62bd-4672-4367-a6df-55b3a23b13c2" containerID="9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344" exitCode=0 Nov 01 21:04:30.618961 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.618804 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" Nov 01 21:04:30.618961 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.618823 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" event={"ID":"760a62bd-4672-4367-a6df-55b3a23b13c2","Type":"ContainerDied","Data":"9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344"} Nov 01 21:04:30.618961 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.618886 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm" event={"ID":"760a62bd-4672-4367-a6df-55b3a23b13c2","Type":"ContainerDied","Data":"826c60bdec9f853f77fe6b77b550f7df3347dafaf6c8196cba0afe95782fc098"} Nov 01 21:04:30.618961 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.618916 4477 scope.go:122] "RemoveContainer" containerID="9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344" Nov 01 21:04:30.628794 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.628776 4477 scope.go:122] "RemoveContainer" containerID="9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344" Nov 01 21:04:30.629067 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:04:30.629045 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344\": container with ID starting with 9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344 not found: ID does not exist" containerID="9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344" Nov 01 21:04:30.629138 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.629094 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344"} err="failed to get container status \"9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344\": rpc error: code = NotFound desc = could not find container \"9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344\": container with ID starting with 9ba0b4c7168f6b788fefa48fece2a183fbcdc8f79ecef90634669a11f597b344 not found: ID does not exist" Nov 01 21:04:30.784610 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.784532 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm"] Nov 01 21:04:30.787594 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:30.787572 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-de8145-5f6f775b8c-n2cgm"] Nov 01 21:04:32.504855 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:32.504822 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="760a62bd-4672-4367-a6df-55b3a23b13c2" path="/var/lib/kubelet/pods/760a62bd-4672-4367-a6df-55b3a23b13c2/volumes" Nov 01 21:04:34.345596 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.345561 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw"] Nov 01 21:04:34.346431 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346405 4477 cpu_manager.go:414] "containerMap: removing container" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" Nov 01 21:04:34.346485 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346435 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" Nov 01 21:04:34.346485 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346466 4477 cpu_manager.go:414] "containerMap: removing container" podUID="29c45515-da77-4ab0-971a-eb97e86c0f4a" containerName="maas-api" Nov 01 21:04:34.346485 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346473 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c45515-da77-4ab0-971a-eb97e86c0f4a" containerName="maas-api" Nov 01 21:04:34.346585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346489 4477 cpu_manager.go:414] "containerMap: removing container" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" Nov 01 21:04:34.346585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346497 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" Nov 01 21:04:34.346585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346517 4477 cpu_manager.go:414] "containerMap: removing container" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" Nov 01 21:04:34.346585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346525 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" Nov 01 21:04:34.346585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346555 4477 cpu_manager.go:414] "containerMap: removing container" podUID="d6c53d18-8bfa-4fd1-8622-41c885aee1fd" containerName="discovery" Nov 01 21:04:34.346585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346562 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c53d18-8bfa-4fd1-8622-41c885aee1fd" containerName="discovery" Nov 01 21:04:34.346585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346573 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" Nov 01 21:04:34.346585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346580 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346600 4477 cpu_manager.go:414] "containerMap: removing container" podUID="760a62bd-4672-4367-a6df-55b3a23b13c2" containerName="maas-api" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346609 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="760a62bd-4672-4367-a6df-55b3a23b13c2" containerName="maas-api" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346727 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="9918869f-25aa-4fbb-ae9f-3881dfd069ca" containerName="istio-proxy" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346743 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c53d18-8bfa-4fd1-8622-41c885aee1fd" containerName="discovery" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346753 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c3b7d1-0ee5-4e04-a1cd-0f071987095d" containerName="istio-proxy" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346765 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="760a62bd-4672-4367-a6df-55b3a23b13c2" containerName="maas-api" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346778 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8790c78d-f8b2-4b0e-8ba6-c180be0b1ecb" containerName="istio-proxy" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346790 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c45515-da77-4ab0-971a-eb97e86c0f4a" containerName="maas-api" Nov 01 21:04:34.346818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.346800 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bfedee-7a85-49a9-9ae4-f607aca1f8ba" containerName="istio-proxy" Nov 01 21:04:34.352173 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.352151 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.434772 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.434738 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.434966 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.434896 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.434966 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.434957 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.435121 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.434996 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.435121 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.435042 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89848dad-3c7a-4bf3-a895-b711c98e2055-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.435238 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.435170 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.435238 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.435219 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.435346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.435257 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9czmr\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-kube-api-access-9czmr\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.435346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.435333 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537108 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537044 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537319 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537177 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537319 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537238 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537319 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537275 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537490 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537320 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89848dad-3c7a-4bf3-a895-b711c98e2055-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537490 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537351 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537490 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537395 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537490 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537434 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czmr\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-kube-api-access-9czmr\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537695 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537494 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537695 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537522 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537695 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537527 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.537876 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.537723 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.538103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.538043 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.538394 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.538368 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89848dad-3c7a-4bf3-a895-b711c98e2055-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.540227 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.540202 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.540591 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.540569 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.569272 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.569235 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw"] Nov 01 21:04:34.593850 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.593817 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.594051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.594028 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9czmr\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-kube-api-access-9czmr\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:34.667415 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:34.667331 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:35.098384 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:04:35.098349 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89848dad_3c7a_4bf3_a895_b711c98e2055.slice/crio-95b167c8e4fe7ff4c34cd81f2a938f7a0e3b36878fac3652010e6c03901aa350 WatchSource:0}: Error finding container 95b167c8e4fe7ff4c34cd81f2a938f7a0e3b36878fac3652010e6c03901aa350: Status 404 returned error can't find the container with id 95b167c8e4fe7ff4c34cd81f2a938f7a0e3b36878fac3652010e6c03901aa350 Nov 01 21:04:35.099034 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:35.098988 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw"] Nov 01 21:04:35.101229 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:35.101189 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:04:35.101326 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:35.101277 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:04:35.101326 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:35.101307 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:04:35.658407 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:35.658374 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" event={"ID":"89848dad-3c7a-4bf3-a895-b711c98e2055","Type":"ContainerStarted","Data":"de459b3f09ae5348e3dc2e60c5d627a91cc2ce23db66159b9d78285a6322ad70"} Nov 01 21:04:35.658823 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:35.658437 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" event={"ID":"89848dad-3c7a-4bf3-a895-b711c98e2055","Type":"ContainerStarted","Data":"95b167c8e4fe7ff4c34cd81f2a938f7a0e3b36878fac3652010e6c03901aa350"} Nov 01 21:04:35.667752 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:35.667727 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:35.793305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:35.793256 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podStartSLOduration=1.793242577 podStartE2EDuration="1.793242577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:35.791598037 +0000 UTC m=+2029.441649583" watchObservedRunningTime="2026-11-01 21:04:35.793242577 +0000 UTC m=+2029.443294121" Nov 01 21:04:36.574751 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:04:36.574716 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:04:36.670197 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:36.670165 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 01 21:04:36.670621 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:36.670215 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 01 21:04:37.667844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:37.667808 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 01 21:04:37.668037 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:37.667876 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 01 21:04:38.668143 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:38.668106 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 01 21:04:38.668577 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:38.668162 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 01 21:04:39.668429 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:39.668392 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 01 21:04:39.668919 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:39.668452 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 01 21:04:40.667654 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:40.667617 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 01 21:04:40.667828 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:40.667669 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 01 21:04:41.505303 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.505269 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8"] Nov 01 21:04:41.505737 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.505489 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" podUID="1370f293-b39b-49da-97e8-92458dcf6e52" containerName="maas-api" containerID="cri-o://ea5577a98ec0cf56c5317ee98c33807a3e0504e918397a0a056e4502b7621730" gracePeriod=30 Nov 01 21:04:41.667815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.667771 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 01 21:04:41.668017 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.667855 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 01 21:04:41.755292 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.754955 4477 generic.go:356] "Generic (PLEG): container finished" podID="1370f293-b39b-49da-97e8-92458dcf6e52" containerID="ea5577a98ec0cf56c5317ee98c33807a3e0504e918397a0a056e4502b7621730" exitCode=0 Nov 01 21:04:41.755292 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.755206 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" event={"ID":"1370f293-b39b-49da-97e8-92458dcf6e52","Type":"ContainerDied","Data":"ea5577a98ec0cf56c5317ee98c33807a3e0504e918397a0a056e4502b7621730"} Nov 01 21:04:41.842893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.842863 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 21:04:41.872564 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.872530 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1370f293-b39b-49da-97e8-92458dcf6e52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370f293-b39b-49da-97e8-92458dcf6e52-openshift-service-ca\") pod \"1370f293-b39b-49da-97e8-92458dcf6e52\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " Nov 01 21:04:41.872780 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.872763 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-metrics-certs\") pod \"1370f293-b39b-49da-97e8-92458dcf6e52\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " Nov 01 21:04:41.872846 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.872832 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls\") pod \"1370f293-b39b-49da-97e8-92458dcf6e52\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " Nov 01 21:04:41.872911 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.872883 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1370f293-b39b-49da-97e8-92458dcf6e52-openshift-service-ca" pod "1370f293-b39b-49da-97e8-92458dcf6e52" (UID: "1370f293-b39b-49da-97e8-92458dcf6e52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:41.872970 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.872902 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1370f293-b39b-49da-97e8-92458dcf6e52-kube-api-access-84j6c\" (UniqueName: \"kubernetes.io/projected/1370f293-b39b-49da-97e8-92458dcf6e52-kube-api-access-84j6c\") pod \"1370f293-b39b-49da-97e8-92458dcf6e52\" (UID: \"1370f293-b39b-49da-97e8-92458dcf6e52\") " Nov 01 21:04:41.873612 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.873500 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370f293-b39b-49da-97e8-92458dcf6e52-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:41.875494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.875458 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls" pod "1370f293-b39b-49da-97e8-92458dcf6e52" (UID: "1370f293-b39b-49da-97e8-92458dcf6e52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:41.875734 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.875706 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-metrics-certs" pod "1370f293-b39b-49da-97e8-92458dcf6e52" (UID: "1370f293-b39b-49da-97e8-92458dcf6e52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:41.876040 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.876020 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1370f293-b39b-49da-97e8-92458dcf6e52-kube-api-access-84j6c" pod "1370f293-b39b-49da-97e8-92458dcf6e52" (UID: "1370f293-b39b-49da-97e8-92458dcf6e52"). InnerVolumeSpecName "kube-api-access-84j6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:41.974831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.974634 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:41.974831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.974665 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1370f293-b39b-49da-97e8-92458dcf6e52-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:41.974831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:41.974682 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84j6c\" (UniqueName: \"kubernetes.io/projected/1370f293-b39b-49da-97e8-92458dcf6e52-kube-api-access-84j6c\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:42.693284 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:42.693251 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:42.694600 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:42.694568 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:42.696000 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:42.695981 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:04:42.763447 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:42.763365 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" Nov 01 21:04:42.763447 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:42.763414 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8" event={"ID":"1370f293-b39b-49da-97e8-92458dcf6e52","Type":"ContainerDied","Data":"ef7e1efdc5e77d644ca20680037e7829aadf4b5cd59d7c22782cd58f41e0b43b"} Nov 01 21:04:42.763693 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:42.763480 4477 scope.go:122] "RemoveContainer" containerID="ea5577a98ec0cf56c5317ee98c33807a3e0504e918397a0a056e4502b7621730" Nov 01 21:04:42.973381 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:42.973348 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8"] Nov 01 21:04:42.976998 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:42.976973 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9b439-68494947fb-mk2x8"] Nov 01 21:04:44.504655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:44.504619 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1370f293-b39b-49da-97e8-92458dcf6e52" path="/var/lib/kubelet/pods/1370f293-b39b-49da-97e8-92458dcf6e52/volumes" Nov 01 21:04:52.928928 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:52.928888 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b"] Nov 01 21:04:52.929469 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:52.929186 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" podUID="26f1dd72-fa2d-4a16-bbde-2199c9422788" containerName="maas-api" containerID="cri-o://6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9" gracePeriod=30 Nov 01 21:04:53.281715 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.281686 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:04:53.344288 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.344248 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f1dd72-fa2d-4a16-bbde-2199c9422788-kube-api-access-2rvsv\" (UniqueName: \"kubernetes.io/projected/26f1dd72-fa2d-4a16-bbde-2199c9422788-kube-api-access-2rvsv\") pod \"26f1dd72-fa2d-4a16-bbde-2199c9422788\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " Nov 01 21:04:53.344506 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.344356 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-metrics-certs\") pod \"26f1dd72-fa2d-4a16-bbde-2199c9422788\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " Nov 01 21:04:53.344506 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.344407 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls\") pod \"26f1dd72-fa2d-4a16-bbde-2199c9422788\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " Nov 01 21:04:53.344662 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.344637 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f1dd72-fa2d-4a16-bbde-2199c9422788-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f1dd72-fa2d-4a16-bbde-2199c9422788-openshift-service-ca\") pod \"26f1dd72-fa2d-4a16-bbde-2199c9422788\" (UID: \"26f1dd72-fa2d-4a16-bbde-2199c9422788\") " Nov 01 21:04:53.345311 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.345276 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f1dd72-fa2d-4a16-bbde-2199c9422788-openshift-service-ca" pod "26f1dd72-fa2d-4a16-bbde-2199c9422788" (UID: "26f1dd72-fa2d-4a16-bbde-2199c9422788"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:53.347775 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.347740 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f1dd72-fa2d-4a16-bbde-2199c9422788-kube-api-access-2rvsv" pod "26f1dd72-fa2d-4a16-bbde-2199c9422788" (UID: "26f1dd72-fa2d-4a16-bbde-2199c9422788"). InnerVolumeSpecName "kube-api-access-2rvsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:53.347879 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.347815 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls" pod "26f1dd72-fa2d-4a16-bbde-2199c9422788" (UID: "26f1dd72-fa2d-4a16-bbde-2199c9422788"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:53.347879 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.347855 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-metrics-certs" pod "26f1dd72-fa2d-4a16-bbde-2199c9422788" (UID: "26f1dd72-fa2d-4a16-bbde-2199c9422788"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:53.451103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.446477 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:53.451103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.446518 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f1dd72-fa2d-4a16-bbde-2199c9422788-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:53.451103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.446535 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f1dd72-fa2d-4a16-bbde-2199c9422788-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:53.451103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.446550 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvsv\" (UniqueName: \"kubernetes.io/projected/26f1dd72-fa2d-4a16-bbde-2199c9422788-kube-api-access-2rvsv\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:53.868462 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.868424 4477 generic.go:356] "Generic (PLEG): container finished" podID="26f1dd72-fa2d-4a16-bbde-2199c9422788" containerID="6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9" exitCode=0 Nov 01 21:04:53.868657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.868475 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" Nov 01 21:04:53.868657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.868508 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" event={"ID":"26f1dd72-fa2d-4a16-bbde-2199c9422788","Type":"ContainerDied","Data":"6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9"} Nov 01 21:04:53.868657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.868560 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b" event={"ID":"26f1dd72-fa2d-4a16-bbde-2199c9422788","Type":"ContainerDied","Data":"c74f418abd2a83e88680e73ee3cd20818319aaaa212a771090b6cccbab605ff2"} Nov 01 21:04:53.868657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.868581 4477 scope.go:122] "RemoveContainer" containerID="6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9" Nov 01 21:04:53.879840 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.879823 4477 scope.go:122] "RemoveContainer" containerID="6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9" Nov 01 21:04:53.880112 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:04:53.880093 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9\": container with ID starting with 6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9 not found: ID does not exist" containerID="6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9" Nov 01 21:04:53.880195 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:53.880122 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9"} err="failed to get container status \"6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9\": rpc error: code = NotFound desc = could not find container \"6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9\": container with ID starting with 6d3cdc48fe8b125ea28a29ef6660c09c09e792cda40851bcbcc6a725ac5404f9 not found: ID does not exist" Nov 01 21:04:54.047099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:54.046477 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b"] Nov 01 21:04:54.053343 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:54.053310 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-081c8e-67f575fb99-n9z6b"] Nov 01 21:04:54.506668 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:04:54.506637 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f1dd72-fa2d-4a16-bbde-2199c9422788" path="/var/lib/kubelet/pods/26f1dd72-fa2d-4a16-bbde-2199c9422788/volumes" Nov 01 21:05:06.559208 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:05:06.559161 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:05:11.311352 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:11.311321 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:05:16.020957 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:16.020914 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm"] Nov 01 21:05:16.070564 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:16.070525 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv"] Nov 01 21:05:31.045375 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.045309 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" containerID="cri-o://97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253" gracePeriod=45 Nov 01 21:05:31.090755 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.090712 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": dial tcp 10.133.0.80:8000: connect: connection refused" Nov 01 21:05:31.092269 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.092211 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" containerID="cri-o://35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48" gracePeriod=45 Nov 01 21:05:31.106277 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:05:31.106240 4477 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2fefb38_b099_4c25_a5ec_1934adac9046.slice/crio-conmon-97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:05:31.376752 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.376730 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 21:05:31.381155 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.381137 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484183 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-tmp-dir\") pod \"b2fefb38-b099-4c25-a5ec-1934adac9046\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484258 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-model-cache\") pod \"ab94912a-4370-4155-965b-e4661b7a75d2\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484291 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fefb38-b099-4c25-a5ec-1934adac9046-kube-api-access-mkbbp\" (UniqueName: \"kubernetes.io/projected/b2fefb38-b099-4c25-a5ec-1934adac9046-kube-api-access-mkbbp\") pod \"b2fefb38-b099-4c25-a5ec-1934adac9046\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484343 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab94912a-4370-4155-965b-e4661b7a75d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94912a-4370-4155-965b-e4661b7a75d2-tls-certs\") pod \"ab94912a-4370-4155-965b-e4661b7a75d2\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484392 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-dshm\") pod \"b2fefb38-b099-4c25-a5ec-1934adac9046\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484432 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2fefb38-b099-4c25-a5ec-1934adac9046-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2fefb38-b099-4c25-a5ec-1934adac9046-tls-certs\") pod \"b2fefb38-b099-4c25-a5ec-1934adac9046\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484478 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab94912a-4370-4155-965b-e4661b7a75d2-kube-api-access-j7v85\" (UniqueName: \"kubernetes.io/projected/ab94912a-4370-4155-965b-e4661b7a75d2-kube-api-access-j7v85\") pod \"ab94912a-4370-4155-965b-e4661b7a75d2\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484553 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-dshm\") pod \"ab94912a-4370-4155-965b-e4661b7a75d2\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484697 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-home\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-home\") pod \"ab94912a-4370-4155-965b-e4661b7a75d2\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484742 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-model-cache\") pod \"b2fefb38-b099-4c25-a5ec-1934adac9046\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484833 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-tmp-dir\") pod \"ab94912a-4370-4155-965b-e4661b7a75d2\" (UID: \"ab94912a-4370-4155-965b-e4661b7a75d2\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.484877 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-home\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-home\") pod \"b2fefb38-b099-4c25-a5ec-1934adac9046\" (UID: \"b2fefb38-b099-4c25-a5ec-1934adac9046\") " Nov 01 21:05:31.485456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.485394 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-home" pod "ab94912a-4370-4155-965b-e4661b7a75d2" (UID: "ab94912a-4370-4155-965b-e4661b7a75d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.486314 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.485718 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-model-cache" pod "b2fefb38-b099-4c25-a5ec-1934adac9046" (UID: "b2fefb38-b099-4c25-a5ec-1934adac9046"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.486314 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.485918 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-tmp-dir" pod "ab94912a-4370-4155-965b-e4661b7a75d2" (UID: "ab94912a-4370-4155-965b-e4661b7a75d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.486440 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.486391 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-tmp-dir" pod "b2fefb38-b099-4c25-a5ec-1934adac9046" (UID: "b2fefb38-b099-4c25-a5ec-1934adac9046"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.486517 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.486497 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-model-cache" pod "ab94912a-4370-4155-965b-e4661b7a75d2" (UID: "ab94912a-4370-4155-965b-e4661b7a75d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.486956 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.486932 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-home" pod "b2fefb38-b099-4c25-a5ec-1934adac9046" (UID: "b2fefb38-b099-4c25-a5ec-1934adac9046"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.488405 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.488176 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2fefb38-b099-4c25-a5ec-1934adac9046-tls-certs" pod "b2fefb38-b099-4c25-a5ec-1934adac9046" (UID: "b2fefb38-b099-4c25-a5ec-1934adac9046"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:05:31.489065 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.489044 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-dshm" pod "ab94912a-4370-4155-965b-e4661b7a75d2" (UID: "ab94912a-4370-4155-965b-e4661b7a75d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.490883 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.490792 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fefb38-b099-4c25-a5ec-1934adac9046-kube-api-access-mkbbp" pod "b2fefb38-b099-4c25-a5ec-1934adac9046" (UID: "b2fefb38-b099-4c25-a5ec-1934adac9046"). InnerVolumeSpecName "kube-api-access-mkbbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:31.491064 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.491040 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-dshm" pod "b2fefb38-b099-4c25-a5ec-1934adac9046" (UID: "b2fefb38-b099-4c25-a5ec-1934adac9046"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.491193 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.491170 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab94912a-4370-4155-965b-e4661b7a75d2-tls-certs" pod "ab94912a-4370-4155-965b-e4661b7a75d2" (UID: "ab94912a-4370-4155-965b-e4661b7a75d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:05:31.491787 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.491631 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab94912a-4370-4155-965b-e4661b7a75d2-kube-api-access-j7v85" pod "ab94912a-4370-4155-965b-e4661b7a75d2" (UID: "ab94912a-4370-4155-965b-e4661b7a75d2"). InnerVolumeSpecName "kube-api-access-j7v85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:31.586994 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.586957 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.586994 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.586990 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.586994 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.586999 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.586994 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587008 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.587387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587019 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkbbp\" (UniqueName: \"kubernetes.io/projected/b2fefb38-b099-4c25-a5ec-1934adac9046-kube-api-access-mkbbp\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.587387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587029 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94912a-4370-4155-965b-e4661b7a75d2-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.587387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587037 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.587387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587045 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2fefb38-b099-4c25-a5ec-1934adac9046-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.587387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587053 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7v85\" (UniqueName: \"kubernetes.io/projected/ab94912a-4370-4155-965b-e4661b7a75d2-kube-api-access-j7v85\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.587387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587061 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.587387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587070 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab94912a-4370-4155-965b-e4661b7a75d2-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.587387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:31.587109 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2fefb38-b099-4c25-a5ec-1934adac9046-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:32.174469 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.174430 4477 generic.go:356] "Generic (PLEG): container finished" podID="ab94912a-4370-4155-965b-e4661b7a75d2" containerID="35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48" exitCode=0 Nov 01 21:05:32.174948 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.174493 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" Nov 01 21:05:32.174948 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.174512 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" event={"ID":"ab94912a-4370-4155-965b-e4661b7a75d2","Type":"ContainerDied","Data":"35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48"} Nov 01 21:05:32.174948 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.174554 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv" event={"ID":"ab94912a-4370-4155-965b-e4661b7a75d2","Type":"ContainerDied","Data":"c31b1b61bb1e369e6aa9bc15aca5a758834c0c9a2e5b2df9d80c170f32d6a0a1"} Nov 01 21:05:32.174948 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.174571 4477 scope.go:122] "RemoveContainer" containerID="35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48" Nov 01 21:05:32.176107 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.176054 4477 generic.go:356] "Generic (PLEG): container finished" podID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerID="97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253" exitCode=0 Nov 01 21:05:32.176204 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.176107 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" event={"ID":"b2fefb38-b099-4c25-a5ec-1934adac9046","Type":"ContainerDied","Data":"97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253"} Nov 01 21:05:32.176204 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.176139 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" event={"ID":"b2fefb38-b099-4c25-a5ec-1934adac9046","Type":"ContainerDied","Data":"4a5eb2d67b761222e0022a93de2a3dc66a06e43a08b230fc334fd341ff3fe3cb"} Nov 01 21:05:32.176204 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.176143 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm" Nov 01 21:05:32.185608 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.185232 4477 scope.go:122] "RemoveContainer" containerID="35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48" Nov 01 21:05:32.185608 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:05:32.185548 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48\": container with ID starting with 35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48 not found: ID does not exist" containerID="35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48" Nov 01 21:05:32.185726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.185611 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48"} err="failed to get container status \"35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48\": rpc error: code = NotFound desc = could not find container \"35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48\": container with ID starting with 35cea3bf9b99106e4298ba1b52efaa91b33049f7b6e1737cdffe940a1b62fa48 not found: ID does not exist" Nov 01 21:05:32.185726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.185630 4477 scope.go:122] "RemoveContainer" containerID="97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253" Nov 01 21:05:32.194763 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.194747 4477 scope.go:122] "RemoveContainer" containerID="97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253" Nov 01 21:05:32.195032 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:05:32.195006 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253\": container with ID starting with 97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253 not found: ID does not exist" containerID="97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253" Nov 01 21:05:32.195089 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.195044 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253"} err="failed to get container status \"97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253\": rpc error: code = NotFound desc = could not find container \"97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253\": container with ID starting with 97de8afcb063b1ee7defb2564fc6c15991e42a43391c7814b56675e7a115e253 not found: ID does not exist" Nov 01 21:05:32.339102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.339058 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv"] Nov 01 21:05:32.388103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.388065 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-w5-c9ae21-kserve-595b75d75f-wjtdv"] Nov 01 21:05:32.449832 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.449749 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm"] Nov 01 21:05:32.452818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.452793 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/distinct-2-w5-c9ae21-kserve-9887c5868-gh6cm"] Nov 01 21:05:32.505069 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.505040 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" path="/var/lib/kubelet/pods/ab94912a-4370-4155-965b-e4661b7a75d2/volumes" Nov 01 21:05:32.505501 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:32.505487 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" path="/var/lib/kubelet/pods/b2fefb38-b099-4c25-a5ec-1934adac9046/volumes" Nov 01 21:05:36.563039 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:05:36.562992 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:05:58.474494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.474396 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4"] Nov 01 21:05:58.475250 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475230 4477 cpu_manager.go:414] "containerMap: removing container" podUID="26f1dd72-fa2d-4a16-bbde-2199c9422788" containerName="maas-api" Nov 01 21:05:58.475312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475254 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f1dd72-fa2d-4a16-bbde-2199c9422788" containerName="maas-api" Nov 01 21:05:58.475312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475279 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" Nov 01 21:05:58.475312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475287 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" Nov 01 21:05:58.475312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475305 4477 cpu_manager.go:414] "containerMap: removing container" podUID="1370f293-b39b-49da-97e8-92458dcf6e52" containerName="maas-api" Nov 01 21:05:58.475438 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475314 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1370f293-b39b-49da-97e8-92458dcf6e52" containerName="maas-api" Nov 01 21:05:58.475438 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475334 4477 cpu_manager.go:414] "containerMap: removing container" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" Nov 01 21:05:58.475438 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475341 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" Nov 01 21:05:58.475529 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475493 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f1dd72-fa2d-4a16-bbde-2199c9422788" containerName="maas-api" Nov 01 21:05:58.475529 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475509 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2fefb38-b099-4c25-a5ec-1934adac9046" containerName="main" Nov 01 21:05:58.475529 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475520 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="1370f293-b39b-49da-97e8-92458dcf6e52" containerName="maas-api" Nov 01 21:05:58.475623 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.475530 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab94912a-4370-4155-965b-e4661b7a75d2" containerName="main" Nov 01 21:05:58.480576 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.480556 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.557027 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.556995 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4ac1ef-openshift-default-dockercfg-cmhjm\"" Nov 01 21:05:58.631147 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.623867 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4"] Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.641844 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-istio-token\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.641899 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-credential-socket\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.641999 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-socket\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.642022 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-certs\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.642047 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c13059-d000-41e5-90af-2536921d5165-istiod-ca-cert\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.642192 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c13059-d000-41e5-90af-2536921d5165-istio-podinfo\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.642251 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2svs7\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-kube-api-access-2svs7\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.642292 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-envoy\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.645097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.642340 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-data\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.743714 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.743611 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c13059-d000-41e5-90af-2536921d5165-istio-podinfo\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.743714 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.743705 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2svs7\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-kube-api-access-2svs7\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.743967 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.743748 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-envoy\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.743967 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.743796 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-data\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.743967 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.743855 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-istio-token\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.743967 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.743882 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-credential-socket\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.744231 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.743991 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-socket\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.744231 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.744014 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-certs\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.744231 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.744040 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c13059-d000-41e5-90af-2536921d5165-istiod-ca-cert\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.744397 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.744308 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-data\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.744919 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.744664 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-socket\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.744919 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.744806 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-certs\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.744919 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.744821 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-credential-socket\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.745186 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.744954 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c13059-d000-41e5-90af-2536921d5165-istiod-ca-cert\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.746429 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.746404 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-envoy\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.747291 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.747269 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c13059-d000-41e5-90af-2536921d5165-istio-podinfo\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.800910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.800881 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2svs7\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-kube-api-access-2svs7\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:58.801143 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:58.801116 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-istio-token\") pod \"e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:59.099225 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:59.099191 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:05:59.535307 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:59.535216 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4"] Nov 01 21:05:59.576157 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:05:59.576131 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c13059_d000_41e5_90af_2536921d5165.slice/crio-735c01731e8270e0cc48e3a7aa8856f76c533e725659028d6fd89bdeb6cb2ab9 WatchSource:0}: Error finding container 735c01731e8270e0cc48e3a7aa8856f76c533e725659028d6fd89bdeb6cb2ab9: Status 404 returned error can't find the container with id 735c01731e8270e0cc48e3a7aa8856f76c533e725659028d6fd89bdeb6cb2ab9 Nov 01 21:05:59.578703 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:59.578669 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:05:59.578775 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:59.578753 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:05:59.578817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:05:59.578782 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:06:00.384492 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:00.384454 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" event={"ID":"b6c13059-d000-41e5-90af-2536921d5165","Type":"ContainerStarted","Data":"2f7d0e6fc5fbbd5eaeb9982ddb2668bde088c5b33fc3978a3970555655db097f"} Nov 01 21:06:00.384700 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:00.384513 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" event={"ID":"b6c13059-d000-41e5-90af-2536921d5165","Type":"ContainerStarted","Data":"735c01731e8270e0cc48e3a7aa8856f76c533e725659028d6fd89bdeb6cb2ab9"} Nov 01 21:06:00.555718 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:00.555670 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" podStartSLOduration=2.555655904 podStartE2EDuration="2.555655904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:00.550976834 +0000 UTC m=+2114.201028381" watchObservedRunningTime="2026-11-01 21:06:00.555655904 +0000 UTC m=+2114.205707449" Nov 01 21:06:01.100056 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:01.100022 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:06:01.105726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:01.105696 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:06:01.392395 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:01.392307 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:06:01.393215 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:01.393199 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:06:06.553716 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:06:06.553682 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:06:08.696974 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:08.696936 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h"] Nov 01 21:06:08.697535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:08.697228 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" podUID="8fb70160-bb70-4ebf-b279-8f792298d96b" containerName="maas-api" containerID="cri-o://fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5" gracePeriod=30 Nov 01 21:06:09.033062 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.033033 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:06:09.104102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.103087 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb70160-bb70-4ebf-b279-8f792298d96b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb70160-bb70-4ebf-b279-8f792298d96b-openshift-service-ca\") pod \"8fb70160-bb70-4ebf-b279-8f792298d96b\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " Nov 01 21:06:09.104102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.103216 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-maas-api-tls\") pod \"8fb70160-bb70-4ebf-b279-8f792298d96b\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " Nov 01 21:06:09.104102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.103271 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-metrics-certs\") pod \"8fb70160-bb70-4ebf-b279-8f792298d96b\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " Nov 01 21:06:09.104102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.103297 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb70160-bb70-4ebf-b279-8f792298d96b-kube-api-access-qgfhl\" (UniqueName: \"kubernetes.io/projected/8fb70160-bb70-4ebf-b279-8f792298d96b-kube-api-access-qgfhl\") pod \"8fb70160-bb70-4ebf-b279-8f792298d96b\" (UID: \"8fb70160-bb70-4ebf-b279-8f792298d96b\") " Nov 01 21:06:09.104520 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.104489 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb70160-bb70-4ebf-b279-8f792298d96b-openshift-service-ca" pod "8fb70160-bb70-4ebf-b279-8f792298d96b" (UID: "8fb70160-bb70-4ebf-b279-8f792298d96b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:09.123597 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.123333 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb70160-bb70-4ebf-b279-8f792298d96b-kube-api-access-qgfhl" pod "8fb70160-bb70-4ebf-b279-8f792298d96b" (UID: "8fb70160-bb70-4ebf-b279-8f792298d96b"). InnerVolumeSpecName "kube-api-access-qgfhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:09.127683 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.127651 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-maas-api-tls" pod "8fb70160-bb70-4ebf-b279-8f792298d96b" (UID: "8fb70160-bb70-4ebf-b279-8f792298d96b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:06:09.128267 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.128228 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-metrics-certs" pod "8fb70160-bb70-4ebf-b279-8f792298d96b" (UID: "8fb70160-bb70-4ebf-b279-8f792298d96b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:06:09.207128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.207064 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb70160-bb70-4ebf-b279-8f792298d96b-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:09.207128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.207125 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:09.207381 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.207143 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb70160-bb70-4ebf-b279-8f792298d96b-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:09.207381 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.207158 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgfhl\" (UniqueName: \"kubernetes.io/projected/8fb70160-bb70-4ebf-b279-8f792298d96b-kube-api-access-qgfhl\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:09.452272 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.452237 4477 generic.go:356] "Generic (PLEG): container finished" podID="8fb70160-bb70-4ebf-b279-8f792298d96b" containerID="fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5" exitCode=0 Nov 01 21:06:09.452498 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.452296 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" Nov 01 21:06:09.452498 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.452326 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" event={"ID":"8fb70160-bb70-4ebf-b279-8f792298d96b","Type":"ContainerDied","Data":"fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5"} Nov 01 21:06:09.452498 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.452374 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h" event={"ID":"8fb70160-bb70-4ebf-b279-8f792298d96b","Type":"ContainerDied","Data":"1e50bf719665d2764c03f3bcedf410ee7ad23c34577ba6e6635c632b39bcfe4d"} Nov 01 21:06:09.452498 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.452395 4477 scope.go:122] "RemoveContainer" containerID="fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5" Nov 01 21:06:09.464839 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.464819 4477 scope.go:122] "RemoveContainer" containerID="fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5" Nov 01 21:06:09.465138 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:06:09.465114 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5\": container with ID starting with fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5 not found: ID does not exist" containerID="fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5" Nov 01 21:06:09.465223 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.465147 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5"} err="failed to get container status \"fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5\": rpc error: code = NotFound desc = could not find container \"fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5\": container with ID starting with fb43f23cd54e3f263bbb5cec729aae225ef4282858c05cc6106accb4e369f7a5 not found: ID does not exist" Nov 01 21:06:09.620382 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.620348 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h"] Nov 01 21:06:09.670762 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:09.670730 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29baa8-6d7bb76468-vkn6h"] Nov 01 21:06:10.504503 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:10.504468 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb70160-bb70-4ebf-b279-8f792298d96b" path="/var/lib/kubelet/pods/8fb70160-bb70-4ebf-b279-8f792298d96b/volumes" Nov 01 21:06:17.224404 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:17.224365 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:06:19.936980 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:19.936943 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf"] Nov 01 21:06:19.937543 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:19.937516 4477 cpu_manager.go:414] "containerMap: removing container" podUID="8fb70160-bb70-4ebf-b279-8f792298d96b" containerName="maas-api" Nov 01 21:06:19.937543 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:19.937530 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb70160-bb70-4ebf-b279-8f792298d96b" containerName="maas-api" Nov 01 21:06:19.937669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:19.937613 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb70160-bb70-4ebf-b279-8f792298d96b" containerName="maas-api" Nov 01 21:06:19.942294 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:19.942272 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:19.995893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:19.995859 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-deff5e-openshift-default-dockercfg-zgcpb\"" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077112 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-certs\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077173 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dea6272-9dbc-435f-a85d-721b92bc7302-istiod-ca-cert\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077201 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-podinfo\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077274 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-credential-socket\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077332 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-envoy\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077401 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-socket\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077480 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-token\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077534 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-data\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.077754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.077611 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjpfj\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-kube-api-access-cjpfj\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.094817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.094680 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf"] Nov 01 21:06:20.179033 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.178995 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-envoy\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179259 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179069 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-socket\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179259 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179198 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-token\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179259 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179249 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-data\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179481 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179319 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjpfj\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-kube-api-access-cjpfj\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179481 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179387 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-certs\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179481 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179421 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dea6272-9dbc-435f-a85d-721b92bc7302-istiod-ca-cert\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179481 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179445 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-podinfo\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179695 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179516 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-credential-socket\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179695 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179576 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-socket\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179801 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179714 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-data\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179901 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179876 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-credential-socket\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.179966 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.179938 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-certs\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.180221 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.180190 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dea6272-9dbc-435f-a85d-721b92bc7302-istiod-ca-cert\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.182116 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.182090 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-envoy\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.182355 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.182334 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-podinfo\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.236938 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.236860 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-token\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.237160 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.237141 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjpfj\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-kube-api-access-cjpfj\") pod \"e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.265765 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.265738 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:20.709740 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.709710 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf"] Nov 01 21:06:20.711131 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:06:20.711101 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dea6272_9dbc_435f_a85d_721b92bc7302.slice/crio-91a2ba4a987b2870e81d629b85e2182445e1db3db04810929c09940e4dfca459 WatchSource:0}: Error finding container 91a2ba4a987b2870e81d629b85e2182445e1db3db04810929c09940e4dfca459: Status 404 returned error can't find the container with id 91a2ba4a987b2870e81d629b85e2182445e1db3db04810929c09940e4dfca459 Nov 01 21:06:20.713759 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.713722 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:06:20.713869 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.713790 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:06:20.713869 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:20.713827 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:06:21.548175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:21.548134 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" event={"ID":"4dea6272-9dbc-435f-a85d-721b92bc7302","Type":"ContainerStarted","Data":"f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69"} Nov 01 21:06:21.548625 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:21.548184 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" event={"ID":"4dea6272-9dbc-435f-a85d-721b92bc7302","Type":"ContainerStarted","Data":"91a2ba4a987b2870e81d629b85e2182445e1db3db04810929c09940e4dfca459"} Nov 01 21:06:21.713323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:21.713246 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" podStartSLOduration=2.713228387 podStartE2EDuration="2.713228387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:21.712486895 +0000 UTC m=+2135.362538457" watchObservedRunningTime="2026-11-01 21:06:21.713228387 +0000 UTC m=+2135.363279944" Nov 01 21:06:22.266896 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:22.266854 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:22.272901 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:22.272875 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:22.555584 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:22.555503 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:22.556575 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:22.556557 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:06:23.696004 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:23.695964 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j"] Nov 01 21:06:23.696511 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:23.696271 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" containerID="cri-o://5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6" gracePeriod=30 Nov 01 21:06:28.983953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:28.983929 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:06:29.014398 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014370 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-credential-socket\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014536 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014406 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-kube-api-access-26w9h\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-kube-api-access-26w9h\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014536 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014427 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-envoy\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014536 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014469 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-data\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014702 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014538 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-podinfo\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014702 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014574 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-token\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-token\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014702 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014666 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-socket\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014702 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014676 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-credential-socket" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:29.014922 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014705 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-certs\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014922 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014759 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f81586-ea85-4a06-a08e-66d6f186a24a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f81586-ea85-4a06-a08e-66d6f186a24a-istiod-ca-cert\") pod \"b8f81586-ea85-4a06-a08e-66d6f186a24a\" (UID: \"b8f81586-ea85-4a06-a08e-66d6f186a24a\") " Nov 01 21:06:29.014922 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.014845 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-data" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:29.015107 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.015045 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-socket" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:29.015595 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.015389 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.015595 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.015413 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.015595 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.015427 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.015595 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.015547 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f81586-ea85-4a06-a08e-66d6f186a24a-istiod-ca-cert" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:29.015860 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.015563 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-certs" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:29.017199 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.017178 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-token" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:29.017294 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.017212 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-kube-api-access-26w9h" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "kube-api-access-26w9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:29.017294 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.017229 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-envoy" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:29.017669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.017642 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-podinfo" pod "b8f81586-ea85-4a06-a08e-66d6f186a24a" (UID: "b8f81586-ea85-4a06-a08e-66d6f186a24a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:06:29.117055 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.117018 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26w9h\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-kube-api-access-26w9h\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.117055 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.117048 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.117055 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.117058 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.117055 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.117067 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f81586-ea85-4a06-a08e-66d6f186a24a-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.117348 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.117090 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f81586-ea85-4a06-a08e-66d6f186a24a-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.117348 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.117100 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f81586-ea85-4a06-a08e-66d6f186a24a-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:29.607967 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.607932 4477 generic.go:356] "Generic (PLEG): container finished" podID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerID="5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6" exitCode=0 Nov 01 21:06:29.608156 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.607989 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" Nov 01 21:06:29.608156 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.608013 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" event={"ID":"b8f81586-ea85-4a06-a08e-66d6f186a24a","Type":"ContainerDied","Data":"5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6"} Nov 01 21:06:29.608156 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.608069 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j" event={"ID":"b8f81586-ea85-4a06-a08e-66d6f186a24a","Type":"ContainerDied","Data":"8129f09608f5766d7205bf160e4d0c4227132478dfa1020e63c6e89c2ac206f9"} Nov 01 21:06:29.608156 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.608097 4477 scope.go:122] "RemoveContainer" containerID="5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6" Nov 01 21:06:29.621421 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.621393 4477 scope.go:122] "RemoveContainer" containerID="ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac" Nov 01 21:06:29.633801 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.633773 4477 scope.go:122] "RemoveContainer" containerID="5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6" Nov 01 21:06:29.634138 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:06:29.634114 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6\": container with ID starting with 5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6 not found: ID does not exist" containerID="5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6" Nov 01 21:06:29.634224 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.634147 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6"} err="failed to get container status \"5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6\": rpc error: code = NotFound desc = could not find container \"5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6\": container with ID starting with 5794eadc7ffe27882f99c884ebb3c0a9a2d3f443a63523751f99e1b83131e4b6 not found: ID does not exist" Nov 01 21:06:29.634224 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.634167 4477 scope.go:122] "RemoveContainer" containerID="ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac" Nov 01 21:06:29.634476 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:06:29.634446 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac\": container with ID starting with ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac not found: ID does not exist" containerID="ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac" Nov 01 21:06:29.634536 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.634481 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac"} err="failed to get container status \"ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac\": rpc error: code = NotFound desc = could not find container \"ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac\": container with ID starting with ddf5512c7a1dd2c24c7f4398258182c9f1ceef6db56115854667d7ad7bcb7bac not found: ID does not exist" Nov 01 21:06:29.771363 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.771330 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j"] Nov 01 21:06:29.821388 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:29.821354 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-22ed7b-openshift-default-7fd8fd8b6f-twm7j"] Nov 01 21:06:30.504732 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:30.504696 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" path="/var/lib/kubelet/pods/b8f81586-ea85-4a06-a08e-66d6f186a24a/volumes" Nov 01 21:06:33.365779 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.365737 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj"] Nov 01 21:06:33.366344 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.366309 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" Nov 01 21:06:33.366344 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.366323 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" Nov 01 21:06:33.366344 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.366345 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" Nov 01 21:06:33.366916 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.366350 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" Nov 01 21:06:33.366916 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.366443 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" Nov 01 21:06:33.366916 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.366462 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f81586-ea85-4a06-a08e-66d6f186a24a" containerName="istio-proxy" Nov 01 21:06:33.371253 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.371230 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.426816 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.426784 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj"] Nov 01 21:06:33.471092 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.471047 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e0a6f5-serving-cert\"" Nov 01 21:06:33.485845 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.485812 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxxb\" (UniqueName: \"kubernetes.io/projected/2416d0e0-25b6-485a-8612-723c279a3e21-kube-api-access-xwxxb\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.485998 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.485955 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.486049 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.486035 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416d0e0-25b6-485a-8612-723c279a3e21-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.486187 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.486172 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-metrics-certs\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.586957 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.586921 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416d0e0-25b6-485a-8612-723c279a3e21-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.587140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.587027 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-metrics-certs\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.587140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.587103 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxxb\" (UniqueName: \"kubernetes.io/projected/2416d0e0-25b6-485a-8612-723c279a3e21-kube-api-access-xwxxb\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.587232 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.587195 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.587337 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:06:33.587317 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-serving-cert: secret "maas-api-e2e-api-a-e0a6f5-serving-cert" not found Nov 01 21:06:33.587418 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:06:33.587398 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls podName:2416d0e0-25b6-485a-8612-723c279a3e21 nodeName:}" failed. No retries permitted until 2026-11-01 21:06:34.087375717 +0000 UTC m=+2147.737427242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls") pod "maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" (UID: "2416d0e0-25b6-485a-8612-723c279a3e21") : secret "maas-api-e2e-api-a-e0a6f5-serving-cert" not found Nov 01 21:06:33.587687 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.587662 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416d0e0-25b6-485a-8612-723c279a3e21-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.589896 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.589870 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-metrics-certs\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:33.644567 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:33.644494 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxxb\" (UniqueName: \"kubernetes.io/projected/2416d0e0-25b6-485a-8612-723c279a3e21-kube-api-access-xwxxb\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:34.095600 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:34.095569 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:34.098142 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:34.098118 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls\") pod \"maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:34.285664 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:34.285626 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:34.664517 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:34.664490 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj"] Nov 01 21:06:34.665322 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:06:34.665294 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2416d0e0_25b6_485a_8612_723c279a3e21.slice/crio-dc48aa1fa25096a28dca1c7f2d033f2d1abe4a71f072a1e6047e31a7ea545ecb WatchSource:0}: Error finding container dc48aa1fa25096a28dca1c7f2d033f2d1abe4a71f072a1e6047e31a7ea545ecb: Status 404 returned error can't find the container with id dc48aa1fa25096a28dca1c7f2d033f2d1abe4a71f072a1e6047e31a7ea545ecb Nov 01 21:06:35.654691 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:35.654641 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" event={"ID":"2416d0e0-25b6-485a-8612-723c279a3e21","Type":"ContainerStarted","Data":"a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968"} Nov 01 21:06:35.654910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:35.654711 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" event={"ID":"2416d0e0-25b6-485a-8612-723c279a3e21","Type":"ContainerStarted","Data":"dc48aa1fa25096a28dca1c7f2d033f2d1abe4a71f072a1e6047e31a7ea545ecb"} Nov 01 21:06:35.654910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:35.654744 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:35.805246 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:35.805136 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" podStartSLOduration=1.959366658 podStartE2EDuration="2.805120929s" totalImagesPullingTime="845.754271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:35.801409285 +0000 UTC m=+2149.451460848" watchObservedRunningTime="2026-11-01 21:06:35.805120929 +0000 UTC m=+2149.455172497" Nov 01 21:06:36.561615 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:06:36.561574 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:06:38.286516 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.286469 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz"] Nov 01 21:06:38.287189 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.286741 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" podUID="c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" containerName="maas-api" containerID="cri-o://ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736" gracePeriod=30 Nov 01 21:06:38.606962 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.606929 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 21:06:38.609655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.609631 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls\") pod \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " Nov 01 21:06:38.609806 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.609682 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-openshift-service-ca\") pod \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " Nov 01 21:06:38.609806 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.609715 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-kube-api-access-pstwt\" (UniqueName: \"kubernetes.io/projected/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-kube-api-access-pstwt\") pod \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " Nov 01 21:06:38.609925 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.609806 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-metrics-certs\") pod \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\" (UID: \"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016\") " Nov 01 21:06:38.610112 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.610065 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-openshift-service-ca" pod "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" (UID: "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:38.610486 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.610466 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:38.612239 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.612214 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-metrics-certs" pod "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" (UID: "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:06:38.612349 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.612263 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-kube-api-access-pstwt" pod "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" (UID: "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016"). InnerVolumeSpecName "kube-api-access-pstwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:38.612490 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.612471 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls" pod "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" (UID: "c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:06:38.675919 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.675880 4477 generic.go:356] "Generic (PLEG): container finished" podID="c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" containerID="ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736" exitCode=0 Nov 01 21:06:38.676152 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.675927 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" Nov 01 21:06:38.676152 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.675931 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" event={"ID":"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016","Type":"ContainerDied","Data":"ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736"} Nov 01 21:06:38.676152 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.676045 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz" event={"ID":"c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016","Type":"ContainerDied","Data":"49bf7353fe1bc47b429f1f73a38999afa9e81737139796026ff52d7bb9b16a2b"} Nov 01 21:06:38.676152 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.676068 4477 scope.go:122] "RemoveContainer" containerID="ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736" Nov 01 21:06:38.689188 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.689136 4477 scope.go:122] "RemoveContainer" containerID="ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736" Nov 01 21:06:38.689903 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:06:38.689879 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736\": container with ID starting with ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736 not found: ID does not exist" containerID="ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736" Nov 01 21:06:38.689989 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.689917 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736"} err="failed to get container status \"ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736\": rpc error: code = NotFound desc = could not find container \"ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736\": container with ID starting with ec394d81f51f410007c8a959ade83ebc6cfff31f29a3343422b99512c520f736 not found: ID does not exist" Nov 01 21:06:38.712108 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.712051 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstwt\" (UniqueName: \"kubernetes.io/projected/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-kube-api-access-pstwt\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:38.712275 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.712115 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:38.712275 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.712132 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:38.833114 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.832918 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz"] Nov 01 21:06:38.836471 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:38.836412 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b27e66-666b979ff7-fv2qz"] Nov 01 21:06:40.504720 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:40.504640 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" path="/var/lib/kubelet/pods/c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016/volumes" Nov 01 21:06:41.667697 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:41.667666 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:06:47.400307 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:06:47.400278 4477 scope.go:122] "RemoveContainer" containerID="049197af55fb5a101c2653090e58aa4f9f8bab71e64955db5db2ad67927340eb" Nov 01 21:07:06.556401 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:07:06.556361 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:07:15.908978 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:15.908927 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt"] Nov 01 21:07:15.910494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:15.910461 4477 cpu_manager.go:414] "containerMap: removing container" podUID="c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" containerName="maas-api" Nov 01 21:07:15.910494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:15.910490 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" containerName="maas-api" Nov 01 21:07:15.910660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:15.910613 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fca6f1-9a04-4a5d-bb8d-718ebe2d8016" containerName="maas-api" Nov 01 21:07:15.914153 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:15.914135 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:15.970203 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:15.970163 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt"] Nov 01 21:07:16.008807 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.008773 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06488271-ae4a-4f4e-8d6f-955a029ca19a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.008991 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.008923 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbv2\" (UniqueName: \"kubernetes.io/projected/06488271-ae4a-4f4e-8d6f-955a029ca19a-kube-api-access-4dbv2\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.008991 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.008977 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.009152 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.009122 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-metrics-certs\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.014633 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.014609 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4ac1ef-serving-cert\"" Nov 01 21:07:16.110905 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.110873 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbv2\" (UniqueName: \"kubernetes.io/projected/06488271-ae4a-4f4e-8d6f-955a029ca19a-kube-api-access-4dbv2\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.111110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.110947 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.111110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.111008 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-metrics-certs\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.111250 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:07:16.111103 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-serving-cert: secret "maas-api-e2e-inf-a-4ac1ef-serving-cert" not found Nov 01 21:07:16.111250 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.111123 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06488271-ae4a-4f4e-8d6f-955a029ca19a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.111250 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:07:16.111168 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls podName:06488271-ae4a-4f4e-8d6f-955a029ca19a nodeName:}" failed. No retries permitted until 2026-11-01 21:07:16.611151563 +0000 UTC m=+2190.261203086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls") pod "maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" (UID: "06488271-ae4a-4f4e-8d6f-955a029ca19a") : secret "maas-api-e2e-inf-a-4ac1ef-serving-cert" not found Nov 01 21:07:16.111764 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.111743 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06488271-ae4a-4f4e-8d6f-955a029ca19a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.113781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.113755 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-metrics-certs\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.170473 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.170377 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbv2\" (UniqueName: \"kubernetes.io/projected/06488271-ae4a-4f4e-8d6f-955a029ca19a-kube-api-access-4dbv2\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.619939 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.619891 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.623106 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.623044 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:16.836053 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:16.836016 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:17.247140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:17.244804 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt"] Nov 01 21:07:17.969624 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:17.969589 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" event={"ID":"06488271-ae4a-4f4e-8d6f-955a029ca19a","Type":"ContainerStarted","Data":"60cc96e58c8b4507cd199019270d9043717e5b95e769d7d5d6ac2bb3d71adecd"} Nov 01 21:07:18.978172 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:18.978138 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" event={"ID":"06488271-ae4a-4f4e-8d6f-955a029ca19a","Type":"ContainerStarted","Data":"524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7"} Nov 01 21:07:18.978579 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:18.978224 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:19.092754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:19.092695 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" podStartSLOduration=3.292400209 podStartE2EDuration="4.092675668s" totalImagesPullingTime="800.275459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:07:19.092208448 +0000 UTC m=+2192.742259984" watchObservedRunningTime="2026-11-01 21:07:19.092675668 +0000 UTC m=+2192.742727214" Nov 01 21:07:23.798350 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:23.798313 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5"] Nov 01 21:07:23.798889 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:23.798596 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" containerID="cri-o://a67b1596a6958ea8d5717ca774c9c04436b7ca1f892a9c36563e2e9f4522c40d" gracePeriod=30 Nov 01 21:07:23.801617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:23.801588 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw"] Nov 01 21:07:23.801877 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:23.801825 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" containerID="cri-o://de459b3f09ae5348e3dc2e60c5d627a91cc2ce23db66159b9d78285a6322ad70" gracePeriod=30 Nov 01 21:07:24.989462 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:24.989426 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:07:26.740989 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:26.740889 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:07:27.697698 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:27.697659 4477 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:07:27.697886 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:27.697723 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:07:29.045057 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.045018 4477 generic.go:356] "Generic (PLEG): container finished" podID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerID="de459b3f09ae5348e3dc2e60c5d627a91cc2ce23db66159b9d78285a6322ad70" exitCode=0 Nov 01 21:07:29.045482 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.045380 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" event={"ID":"89848dad-3c7a-4bf3-a895-b711c98e2055","Type":"ContainerDied","Data":"de459b3f09ae5348e3dc2e60c5d627a91cc2ce23db66159b9d78285a6322ad70"} Nov 01 21:07:29.047554 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.047527 4477 generic.go:356] "Generic (PLEG): container finished" podID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerID="a67b1596a6958ea8d5717ca774c9c04436b7ca1f892a9c36563e2e9f4522c40d" exitCode=0 Nov 01 21:07:29.047692 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.047567 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" event={"ID":"06108661-d8ba-47f9-96ec-cb98a4c2f715","Type":"ContainerDied","Data":"a67b1596a6958ea8d5717ca774c9c04436b7ca1f892a9c36563e2e9f4522c40d"} Nov 01 21:07:29.047692 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.047617 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" event={"ID":"06108661-d8ba-47f9-96ec-cb98a4c2f715","Type":"ContainerDied","Data":"e4f586455a8f7ccb47abfbd5eddbe1af884210e1959ce3f40b076eab33ef10bc"} Nov 01 21:07:29.047692 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.047643 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f586455a8f7ccb47abfbd5eddbe1af884210e1959ce3f40b076eab33ef10bc" Nov 01 21:07:29.047692 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.047660 4477 scope.go:122] "RemoveContainer" containerID="666ef01babc4490ced1e6525ca8b22ffc0111fbd004360adfa84483d1a4d927c" Nov 01 21:07:29.071716 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.071690 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 21:07:29.108849 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.108822 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-socket\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109001 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.108912 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-data\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109001 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.108949 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-envoy\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109141 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109003 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-token\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-token\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109208 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109134 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-kube-api-access-cp9fq\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-kube-api-access-cp9fq\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109208 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109126 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-socket" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.109208 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109194 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-data" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.109340 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109266 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-credential-socket\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109340 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109299 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06108661-d8ba-47f9-96ec-cb98a4c2f715-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108661-d8ba-47f9-96ec-cb98a4c2f715-istiod-ca-cert\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109340 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109330 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-certs\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109502 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109369 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-podinfo\") pod \"06108661-d8ba-47f9-96ec-cb98a4c2f715\" (UID: \"06108661-d8ba-47f9-96ec-cb98a4c2f715\") " Nov 01 21:07:29.109726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109616 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-certs" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.109726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109689 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06108661-d8ba-47f9-96ec-cb98a4c2f715-istiod-ca-cert" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:29.109926 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.109846 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-credential-socket" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.110145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.110049 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.110145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.110094 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.110145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.110109 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.110145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.110120 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108661-d8ba-47f9-96ec-cb98a4c2f715-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.110145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.110128 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.111632 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.111612 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-token" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:29.111724 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.111679 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-podinfo" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:07:29.111893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.111874 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-envoy" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.111893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.111882 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-kube-api-access-cp9fq" pod "06108661-d8ba-47f9-96ec-cb98a4c2f715" (UID: "06108661-d8ba-47f9-96ec-cb98a4c2f715"). InnerVolumeSpecName "kube-api-access-cp9fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:29.114375 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.114358 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:07:29.210832 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.210800 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-credential-socket\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211028 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.210845 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89848dad-3c7a-4bf3-a895-b711c98e2055-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89848dad-3c7a-4bf3-a895-b711c98e2055-istiod-ca-cert\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211028 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.210881 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-certs\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211028 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.210911 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-socket\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211028 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.210986 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-kube-api-access-9czmr\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-kube-api-access-9czmr\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211121 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-token\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-token\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211170 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-data\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211206 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-envoy\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211243 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-podinfo\") pod \"89848dad-3c7a-4bf3-a895-b711c98e2055\" (UID: \"89848dad-3c7a-4bf3-a895-b711c98e2055\") " Nov 01 21:07:29.211512 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211171 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-credential-socket" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.211512 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211206 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-socket" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.211512 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211252 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-certs" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.211512 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211451 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89848dad-3c7a-4bf3-a895-b711c98e2055-istiod-ca-cert" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:29.211512 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211414 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-data" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.211971 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211923 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.211971 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211954 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.211971 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211971 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp9fq\" (UniqueName: \"kubernetes.io/projected/06108661-d8ba-47f9-96ec-cb98a4c2f715-kube-api-access-cp9fq\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.212179 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.211987 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.212179 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.212002 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89848dad-3c7a-4bf3-a895-b711c98e2055-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.212179 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.212067 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.212179 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.212124 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.212332 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.212219 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108661-d8ba-47f9-96ec-cb98a4c2f715-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.212332 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.212233 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.213703 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.213679 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-kube-api-access-9czmr" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "kube-api-access-9czmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:29.213817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.213712 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-envoy" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.214134 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.214110 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-token" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:29.214232 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.214143 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-podinfo" pod "89848dad-3c7a-4bf3-a895-b711c98e2055" (UID: "89848dad-3c7a-4bf3-a895-b711c98e2055"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:07:29.313346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.313309 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9czmr\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-kube-api-access-9czmr\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.313346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.313334 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.313346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.313346 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.313346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:29.313354 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89848dad-3c7a-4bf3-a895-b711c98e2055-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:30.055368 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.055338 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" Nov 01 21:07:30.055368 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.055356 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw" event={"ID":"89848dad-3c7a-4bf3-a895-b711c98e2055","Type":"ContainerDied","Data":"95b167c8e4fe7ff4c34cd81f2a938f7a0e3b36878fac3652010e6c03901aa350"} Nov 01 21:07:30.055861 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.055450 4477 scope.go:122] "RemoveContainer" containerID="de459b3f09ae5348e3dc2e60c5d627a91cc2ce23db66159b9d78285a6322ad70" Nov 01 21:07:30.056928 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.056910 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5" Nov 01 21:07:30.283916 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.283839 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5"] Nov 01 21:07:30.290018 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.289988 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-vx9f5"] Nov 01 21:07:30.397063 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.397032 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw"] Nov 01 21:07:30.399902 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.399882 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-85qkw"] Nov 01 21:07:30.504390 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.504357 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" path="/var/lib/kubelet/pods/06108661-d8ba-47f9-96ec-cb98a4c2f715/volumes" Nov 01 21:07:30.504888 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:30.504872 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" path="/var/lib/kubelet/pods/89848dad-3c7a-4bf3-a895-b711c98e2055/volumes" Nov 01 21:07:36.557325 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:07:36.557289 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:07:40.437753 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.437657 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c"] Nov 01 21:07:40.438872 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.438844 4477 cpu_manager.go:414] "containerMap: removing container" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" Nov 01 21:07:40.438872 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.438872 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" Nov 01 21:07:40.439058 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.438891 4477 cpu_manager.go:414] "containerMap: removing container" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" Nov 01 21:07:40.439058 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.438899 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" Nov 01 21:07:40.439058 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.438951 4477 cpu_manager.go:414] "containerMap: removing container" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" Nov 01 21:07:40.439058 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.438960 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" Nov 01 21:07:40.439290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.439231 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" Nov 01 21:07:40.439290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.439250 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="89848dad-3c7a-4bf3-a895-b711c98e2055" containerName="istio-proxy" Nov 01 21:07:40.444111 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.444065 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.495970 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.495946 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-deff5e-serving-cert\"" Nov 01 21:07:40.549008 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.548985 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c"] Nov 01 21:07:40.551198 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.551174 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzf6t\" (UniqueName: \"kubernetes.io/projected/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-kube-api-access-dzf6t\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.551325 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.551264 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.551445 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.551429 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-openshift-service-ca\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.551513 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.551462 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-metrics-certs\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.652366 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.652330 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-openshift-service-ca\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.652366 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.652369 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-metrics-certs\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.652639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.652433 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzf6t\" (UniqueName: \"kubernetes.io/projected/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-kube-api-access-dzf6t\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.652639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.652511 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.652758 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:07:40.652652 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-serving-cert: secret "maas-api-e2e-api-a-deff5e-serving-cert" not found Nov 01 21:07:40.652758 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:07:40.652709 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls podName:fdeeb41e-d051-4f52-a0d3-787e68a2d19a nodeName:}" failed. No retries permitted until 2026-11-01 21:07:41.152688124 +0000 UTC m=+2214.802739667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls") pod "maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" (UID: "fdeeb41e-d051-4f52-a0d3-787e68a2d19a") : secret "maas-api-e2e-api-a-deff5e-serving-cert" not found Nov 01 21:07:40.653063 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.653041 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-openshift-service-ca\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.655046 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.655011 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-metrics-certs\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:40.710817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:40.710741 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzf6t\" (UniqueName: \"kubernetes.io/projected/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-kube-api-access-dzf6t\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:41.158065 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:41.158026 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:41.160661 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:41.160636 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls\") pod \"maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:41.358976 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:41.358945 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:41.739712 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:41.739672 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c"] Nov 01 21:07:41.741527 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:07:41.741502 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdeeb41e_d051_4f52_a0d3_787e68a2d19a.slice/crio-9990415ce12f83c0a2d46ae0cc480dbb2de8e34759f1a8212e3f71688a7b454b WatchSource:0}: Error finding container 9990415ce12f83c0a2d46ae0cc480dbb2de8e34759f1a8212e3f71688a7b454b: Status 404 returned error can't find the container with id 9990415ce12f83c0a2d46ae0cc480dbb2de8e34759f1a8212e3f71688a7b454b Nov 01 21:07:41.743331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:41.743312 4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:07:42.136172 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:42.136136 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" event={"ID":"fdeeb41e-d051-4f52-a0d3-787e68a2d19a","Type":"ContainerStarted","Data":"9990415ce12f83c0a2d46ae0cc480dbb2de8e34759f1a8212e3f71688a7b454b"} Nov 01 21:07:43.144668 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:43.144618 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" event={"ID":"fdeeb41e-d051-4f52-a0d3-787e68a2d19a","Type":"ContainerStarted","Data":"23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd"} Nov 01 21:07:43.145181 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:43.144697 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:43.269169 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:43.269113 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" podStartSLOduration=2.3646662960000002 podStartE2EDuration="3.269069171s" totalImagesPullingTime="904.402875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:07:43.268118155 +0000 UTC m=+2216.918169699" watchObservedRunningTime="2026-11-01 21:07:43.269069171 +0000 UTC m=+2216.919120726" Nov 01 21:07:47.446018 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:47.445978 4477 scope.go:122] "RemoveContainer" containerID="a67b1596a6958ea8d5717ca774c9c04436b7ca1f892a9c36563e2e9f4522c40d" Nov 01 21:07:49.158653 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:49.158625 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:07:54.841367 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:54.841314 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4"] Nov 01 21:07:54.841810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:54.841625 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" podUID="0347f23c-9e24-472c-b4e0-0da2661b8f6c" containerName="maas-api" containerID="cri-o://b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048" gracePeriod=30 Nov 01 21:07:55.178335 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.178302 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 21:07:55.262797 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.262754 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0347f23c-9e24-472c-b4e0-0da2661b8f6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347f23c-9e24-472c-b4e0-0da2661b8f6c-openshift-service-ca\") pod \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " Nov 01 21:07:55.264311 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.263443 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0347f23c-9e24-472c-b4e0-0da2661b8f6c-openshift-service-ca" pod "0347f23c-9e24-472c-b4e0-0da2661b8f6c" (UID: "0347f23c-9e24-472c-b4e0-0da2661b8f6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:55.264699 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.264674 4477 generic.go:356] "Generic (PLEG): container finished" podID="0347f23c-9e24-472c-b4e0-0da2661b8f6c" containerID="b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048" exitCode=0 Nov 01 21:07:55.265331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.264806 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" event={"ID":"0347f23c-9e24-472c-b4e0-0da2661b8f6c","Type":"ContainerDied","Data":"b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048"} Nov 01 21:07:55.265331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.265021 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" event={"ID":"0347f23c-9e24-472c-b4e0-0da2661b8f6c","Type":"ContainerDied","Data":"0b15d8e8b7f2a68132d8b2693ca0cac1850d12c0f9e84129afa363a163a2a0a0"} Nov 01 21:07:55.265331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.265058 4477 scope.go:122] "RemoveContainer" containerID="b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048" Nov 01 21:07:55.265331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.264876 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4" Nov 01 21:07:55.265961 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.265943 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0347f23c-9e24-472c-b4e0-0da2661b8f6c-kube-api-access-hk8gq\" (UniqueName: \"kubernetes.io/projected/0347f23c-9e24-472c-b4e0-0da2661b8f6c-kube-api-access-hk8gq\") pod \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " Nov 01 21:07:55.267030 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.267010 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-metrics-certs\") pod \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " Nov 01 21:07:55.267667 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.267649 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls\") pod \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\" (UID: \"0347f23c-9e24-472c-b4e0-0da2661b8f6c\") " Nov 01 21:07:55.268652 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.268630 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347f23c-9e24-472c-b4e0-0da2661b8f6c-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:55.272289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.272217 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0347f23c-9e24-472c-b4e0-0da2661b8f6c-kube-api-access-hk8gq" pod "0347f23c-9e24-472c-b4e0-0da2661b8f6c" (UID: "0347f23c-9e24-472c-b4e0-0da2661b8f6c"). InnerVolumeSpecName "kube-api-access-hk8gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:55.289146 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.289093 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls" pod "0347f23c-9e24-472c-b4e0-0da2661b8f6c" (UID: "0347f23c-9e24-472c-b4e0-0da2661b8f6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:07:55.290785 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.290744 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-metrics-certs" pod "0347f23c-9e24-472c-b4e0-0da2661b8f6c" (UID: "0347f23c-9e24-472c-b4e0-0da2661b8f6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:07:55.310364 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.310337 4477 scope.go:122] "RemoveContainer" containerID="b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048" Nov 01 21:07:55.310830 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:07:55.310798 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048\": container with ID starting with b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048 not found: ID does not exist" containerID="b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048" Nov 01 21:07:55.310958 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.310841 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048"} err="failed to get container status \"b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048\": rpc error: code = NotFound desc = could not find container \"b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048\": container with ID starting with b72b28eb5654a778abdb86ceedcdbe089997b98944680bfb592344e76be45048 not found: ID does not exist" Nov 01 21:07:55.369905 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.369857 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk8gq\" (UniqueName: \"kubernetes.io/projected/0347f23c-9e24-472c-b4e0-0da2661b8f6c-kube-api-access-hk8gq\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:55.369905 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.369906 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:55.370232 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.369926 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0347f23c-9e24-472c-b4e0-0da2661b8f6c-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:55.685730 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.685700 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4"] Nov 01 21:07:55.735185 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:55.735153 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9ae21-9dc9c78dc-jh2x4"] Nov 01 21:07:56.504524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:07:56.504494 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0347f23c-9e24-472c-b4e0-0da2661b8f6c" path="/var/lib/kubelet/pods/0347f23c-9e24-472c-b4e0-0da2661b8f6c/volumes" Nov 01 21:08:03.271414 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:03.271379 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8"] Nov 01 21:08:03.271950 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:03.271579 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" containerID="cri-o://7f507be80f83ce139a543634f6a24ffc5846a970a10212ad0b79dde149023ef8" gracePeriod=30 Nov 01 21:08:04.582103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:04.564858 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh"] Nov 01 21:08:04.597099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:04.587646 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" containerID="cri-o://a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05" gracePeriod=30 Nov 01 21:08:06.594051 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:08:06.594013 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:08:08.452889 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.452606 4477 generic.go:356] "Generic (PLEG): container finished" podID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerID="7f507be80f83ce139a543634f6a24ffc5846a970a10212ad0b79dde149023ef8" exitCode=0 Nov 01 21:08:08.452889 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.452774 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" event={"ID":"b30768f6-fbd9-4d98-adcd-59455e939e31","Type":"ContainerDied","Data":"7f507be80f83ce139a543634f6a24ffc5846a970a10212ad0b79dde149023ef8"} Nov 01 21:08:08.577940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.577903 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749248 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-credential-socket\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749312 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-token\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-token\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749345 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-kube-api-access-d52zm\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-kube-api-access-d52zm\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749375 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30768f6-fbd9-4d98-adcd-59455e939e31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30768f6-fbd9-4d98-adcd-59455e939e31-istiod-ca-cert\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749453 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-envoy\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749507 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-data\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749539 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-socket\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749585 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-podinfo\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.750733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.749740 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-certs\") pod \"b30768f6-fbd9-4d98-adcd-59455e939e31\" (UID: \"b30768f6-fbd9-4d98-adcd-59455e939e31\") " Nov 01 21:08:08.752494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.751599 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-data" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:08.752494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.752031 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30768f6-fbd9-4d98-adcd-59455e939e31-istiod-ca-cert" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:08:08.752494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.752281 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-credential-socket" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:08.752745 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.752530 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-socket" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:08.752901 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.752872 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-certs" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:08.767847 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.767798 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-podinfo" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:08:08.774693 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.774638 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-envoy" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:08.774825 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.774681 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-kube-api-access-d52zm" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "kube-api-access-d52zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:08.775019 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.774992 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-token" pod "b30768f6-fbd9-4d98-adcd-59455e939e31" (UID: "b30768f6-fbd9-4d98-adcd-59455e939e31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:08.851770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851731 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:08.851770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851771 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:08.851770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851786 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:08.851770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851802 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d52zm\" (UniqueName: \"kubernetes.io/projected/b30768f6-fbd9-4d98-adcd-59455e939e31-kube-api-access-d52zm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:08.851770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851811 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30768f6-fbd9-4d98-adcd-59455e939e31-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:08.851770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851821 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:08.851770 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851829 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:08.852554 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851837 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30768f6-fbd9-4d98-adcd-59455e939e31-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:08.852554 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:08.851849 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30768f6-fbd9-4d98-adcd-59455e939e31-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:09.461710 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:09.461675 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" Nov 01 21:08:09.461710 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:09.461696 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8" event={"ID":"b30768f6-fbd9-4d98-adcd-59455e939e31","Type":"ContainerDied","Data":"86c19e6ebdccb929fa2a40a1dab617dd84edb28770dcea00fe0e73a9df0e9093"} Nov 01 21:08:09.462305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:09.461811 4477 scope.go:122] "RemoveContainer" containerID="7f507be80f83ce139a543634f6a24ffc5846a970a10212ad0b79dde149023ef8" Nov 01 21:08:09.677175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:09.677148 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8"] Nov 01 21:08:09.726601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:09.726534 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-zjkp8"] Nov 01 21:08:10.341174 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.341145 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 21:08:10.473271 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.473191 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-credential-socket\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.473798 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.473777 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-envoy\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.474003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.473986 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-certs\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.474155 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474140 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-data\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.474304 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474289 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-kube-api-access-ttlr5\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-kube-api-access-ttlr5\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.474419 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474405 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-socket\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.474574 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474559 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istiod-ca-cert\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.474703 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474690 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-token\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-token\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.474835 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474820 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-podinfo\") pod \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\" (UID: \"fd61b1b0-c480-4678-8e78-4e68d4b8b120\") " Nov 01 21:08:10.475972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.473410 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-credential-socket" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:10.475972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474457 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-certs" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:10.475972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474652 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-data" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:10.475972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.474828 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-socket" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:10.475972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.475528 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.475972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.475553 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.475972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.475567 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.475972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.475579 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.476489 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.476009 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istiod-ca-cert" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:08:10.477958 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.477938 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-podinfo" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:08:10.479844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.479808 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-envoy" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:10.480634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.480601 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-token" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:10.480856 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.480825 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-kube-api-access-ttlr5" pod "fd61b1b0-c480-4678-8e78-4e68d4b8b120" (UID: "fd61b1b0-c480-4678-8e78-4e68d4b8b120"). InnerVolumeSpecName "kube-api-access-ttlr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:10.491098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.483542 4477 generic.go:356] "Generic (PLEG): container finished" podID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerID="a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05" exitCode=0 Nov 01 21:08:10.491098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.483658 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" event={"ID":"fd61b1b0-c480-4678-8e78-4e68d4b8b120","Type":"ContainerDied","Data":"a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05"} Nov 01 21:08:10.491098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.483713 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" event={"ID":"fd61b1b0-c480-4678-8e78-4e68d4b8b120","Type":"ContainerDied","Data":"094ceda89bf0e84bf57cbe20794a31fc3ae309f648152720b5338988c11d1c5b"} Nov 01 21:08:10.491098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.483741 4477 scope.go:122] "RemoveContainer" containerID="a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05" Nov 01 21:08:10.491098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.483871 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh" Nov 01 21:08:10.510347 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.509177 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" path="/var/lib/kubelet/pods/b30768f6-fbd9-4d98-adcd-59455e939e31/volumes" Nov 01 21:08:10.521243 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.521221 4477 scope.go:122] "RemoveContainer" containerID="a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05" Nov 01 21:08:10.521943 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:08:10.521900 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05\": container with ID starting with a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05 not found: ID does not exist" containerID="a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05" Nov 01 21:08:10.522045 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.521940 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05"} err="failed to get container status \"a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05\": rpc error: code = NotFound desc = could not find container \"a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05\": container with ID starting with a89675fadae4a183912470a4381e5730bcc32bedd7b9e0c1b4e0ac9aea5e0e05 not found: ID does not exist" Nov 01 21:08:10.577131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.577059 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttlr5\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-kube-api-access-ttlr5\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.577131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.577117 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.577131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.577136 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.577491 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.577150 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.577491 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.577165 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61b1b0-c480-4678-8e78-4e68d4b8b120-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:10.653370 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.653340 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh"] Nov 01 21:08:10.658248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:10.658222 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-qjcxh"] Nov 01 21:08:11.635991 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:11.635964 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz"] Nov 01 21:08:11.636453 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:11.636225 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" podUID="80027a69-7918-4633-b561-d080b6d10131" containerName="maas-api" containerID="cri-o://c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21" gracePeriod=30 Nov 01 21:08:11.927374 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:11.927340 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:08:12.000050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.000010 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-metrics-certs\") pod \"80027a69-7918-4633-b561-d080b6d10131\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " Nov 01 21:08:12.000284 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.000139 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80027a69-7918-4633-b561-d080b6d10131-kube-api-access-q6zmm\" (UniqueName: \"kubernetes.io/projected/80027a69-7918-4633-b561-d080b6d10131-kube-api-access-q6zmm\") pod \"80027a69-7918-4633-b561-d080b6d10131\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " Nov 01 21:08:12.000284 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.000273 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80027a69-7918-4633-b561-d080b6d10131-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80027a69-7918-4633-b561-d080b6d10131-openshift-service-ca\") pod \"80027a69-7918-4633-b561-d080b6d10131\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " Nov 01 21:08:12.000474 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.000350 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-maas-api-tls\") pod \"80027a69-7918-4633-b561-d080b6d10131\" (UID: \"80027a69-7918-4633-b561-d080b6d10131\") " Nov 01 21:08:12.000848 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.000661 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80027a69-7918-4633-b561-d080b6d10131-openshift-service-ca" pod "80027a69-7918-4633-b561-d080b6d10131" (UID: "80027a69-7918-4633-b561-d080b6d10131"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:08:12.000976 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.000880 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80027a69-7918-4633-b561-d080b6d10131-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:12.002729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.002699 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-maas-api-tls" pod "80027a69-7918-4633-b561-d080b6d10131" (UID: "80027a69-7918-4633-b561-d080b6d10131"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:08:12.003037 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.003006 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80027a69-7918-4633-b561-d080b6d10131-kube-api-access-q6zmm" pod "80027a69-7918-4633-b561-d080b6d10131" (UID: "80027a69-7918-4633-b561-d080b6d10131"). InnerVolumeSpecName "kube-api-access-q6zmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:12.003037 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.003014 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-metrics-certs" pod "80027a69-7918-4633-b561-d080b6d10131" (UID: "80027a69-7918-4633-b561-d080b6d10131"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:08:12.105864 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.105826 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:12.105864 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.105859 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80027a69-7918-4633-b561-d080b6d10131-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:12.105864 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.105869 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6zmm\" (UniqueName: \"kubernetes.io/projected/80027a69-7918-4633-b561-d080b6d10131-kube-api-access-q6zmm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:12.500522 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.500489 4477 generic.go:356] "Generic (PLEG): container finished" podID="b6c13059-d000-41e5-90af-2536921d5165" containerID="2f7d0e6fc5fbbd5eaeb9982ddb2668bde088c5b33fc3978a3970555655db097f" exitCode=0 Nov 01 21:08:12.502095 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.502039 4477 generic.go:356] "Generic (PLEG): container finished" podID="80027a69-7918-4633-b561-d080b6d10131" containerID="c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21" exitCode=0 Nov 01 21:08:12.502242 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.502213 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" Nov 01 21:08:12.504056 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.504029 4477 generic.go:356] "Generic (PLEG): container finished" podID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerID="f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69" exitCode=0 Nov 01 21:08:12.504671 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.504651 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" path="/var/lib/kubelet/pods/fd61b1b0-c480-4678-8e78-4e68d4b8b120/volumes" Nov 01 21:08:12.505168 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.505151 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" event={"ID":"b6c13059-d000-41e5-90af-2536921d5165","Type":"ContainerDied","Data":"2f7d0e6fc5fbbd5eaeb9982ddb2668bde088c5b33fc3978a3970555655db097f"} Nov 01 21:08:12.505225 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.505206 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" event={"ID":"80027a69-7918-4633-b561-d080b6d10131","Type":"ContainerDied","Data":"c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21"} Nov 01 21:08:12.505277 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.505233 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz" event={"ID":"80027a69-7918-4633-b561-d080b6d10131","Type":"ContainerDied","Data":"8060b6926afe1b403ebebc83e3e8d337916711e001d7f99d2ae65b51a7bd9780"} Nov 01 21:08:12.505277 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.505255 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" event={"ID":"4dea6272-9dbc-435f-a85d-721b92bc7302","Type":"ContainerDied","Data":"f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69"} Nov 01 21:08:12.505369 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.505317 4477 scope.go:122] "RemoveContainer" containerID="c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21" Nov 01 21:08:12.505691 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.505672 4477 scope.go:122] "RemoveContainer" containerID="2f7d0e6fc5fbbd5eaeb9982ddb2668bde088c5b33fc3978a3970555655db097f" Nov 01 21:08:12.505846 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.505831 4477 scope.go:122] "RemoveContainer" containerID="f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69" Nov 01 21:08:12.510812 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.510647 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:12.511179 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.511152 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:12.511670 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.511645 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:12.513346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.512608 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:12.513823 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.513790 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:12.515029 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.514997 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:12.520415 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.520391 4477 scope.go:122] "RemoveContainer" containerID="c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21" Nov 01 21:08:12.520784 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:08:12.520747 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21\": container with ID starting with c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21 not found: ID does not exist" containerID="c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21" Nov 01 21:08:12.520873 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.520782 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21"} err="failed to get container status \"c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21\": rpc error: code = NotFound desc = could not find container \"c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21\": container with ID starting with c1650be1b3e0078e3a66d8f1f2f1f54f0e2a6feca96147488672187685809d21 not found: ID does not exist" Nov 01 21:08:12.796043 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.795935 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz"] Nov 01 21:08:12.805718 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:12.805692 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d08fd0-74f6487c48-q42pz"] Nov 01 21:08:13.522421 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:13.522382 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" event={"ID":"4dea6272-9dbc-435f-a85d-721b92bc7302","Type":"ContainerStarted","Data":"c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e"} Nov 01 21:08:13.526335 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:13.526297 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" event={"ID":"b6c13059-d000-41e5-90af-2536921d5165","Type":"ContainerStarted","Data":"5a016b859f02558689b53c951ce822343f09f0277e6c1ae12f064cb23fde4d3d"} Nov 01 21:08:14.101207 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:14.101173 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:08:14.102117 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:14.102093 4477 patch_prober.go:29] interesting pod/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 01 21:08:14.102296 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:14.102277 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 01 21:08:14.268140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:14.266519 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:08:14.268140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:14.266648 4477 patch_prober.go:29] interesting pod/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 21:08:14.268140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:14.266690 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 21:08:14.503970 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:14.503898 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80027a69-7918-4633-b561-d080b6d10131" path="/var/lib/kubelet/pods/80027a69-7918-4633-b561-d080b6d10131/volumes" Nov 01 21:08:15.100401 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:15.100367 4477 patch_prober.go:29] interesting pod/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 01 21:08:15.100603 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:15.100425 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 01 21:08:15.271111 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:15.270533 4477 patch_prober.go:29] interesting pod/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 21:08:15.271111 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:15.270617 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 21:08:16.100198 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:16.100161 4477 patch_prober.go:29] interesting pod/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 01 21:08:16.100388 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:16.100227 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 01 21:08:16.274103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:16.273061 4477 patch_prober.go:29] interesting pod/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 21:08:16.274103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:16.273149 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 21:08:16.392377 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:16.392287 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:08:17.099462 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.099430 4477 patch_prober.go:29] interesting pod/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 01 21:08:17.099633 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.099484 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 01 21:08:17.170590 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.170550 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw"] Nov 01 21:08:17.171669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.171641 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" Nov 01 21:08:17.171837 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.171821 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" Nov 01 21:08:17.171971 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.171956 4477 cpu_manager.go:414] "containerMap: removing container" podUID="80027a69-7918-4633-b561-d080b6d10131" containerName="maas-api" Nov 01 21:08:17.172057 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172045 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80027a69-7918-4633-b561-d080b6d10131" containerName="maas-api" Nov 01 21:08:17.172177 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172166 4477 cpu_manager.go:414] "containerMap: removing container" podUID="0347f23c-9e24-472c-b4e0-0da2661b8f6c" containerName="maas-api" Nov 01 21:08:17.172271 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172260 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0347f23c-9e24-472c-b4e0-0da2661b8f6c" containerName="maas-api" Nov 01 21:08:17.172374 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172362 4477 cpu_manager.go:414] "containerMap: removing container" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" Nov 01 21:08:17.172466 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172455 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" Nov 01 21:08:17.172685 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172673 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="06108661-d8ba-47f9-96ec-cb98a4c2f715" containerName="istio-proxy" Nov 01 21:08:17.172776 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172766 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="80027a69-7918-4633-b561-d080b6d10131" containerName="maas-api" Nov 01 21:08:17.172862 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172853 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0347f23c-9e24-472c-b4e0-0da2661b8f6c" containerName="maas-api" Nov 01 21:08:17.172946 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.172937 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd61b1b0-c480-4678-8e78-4e68d4b8b120" containerName="istio-proxy" Nov 01 21:08:17.173029 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.173020 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30768f6-fbd9-4d98-adcd-59455e939e31" containerName="istio-proxy" Nov 01 21:08:17.182981 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.182447 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.267102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.256137 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq"] Nov 01 21:08:17.267102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.259247 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cqvhp\"" Nov 01 21:08:17.274139 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.272493 4477 patch_prober.go:29] interesting pod/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 21:08:17.274139 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.272549 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 21:08:17.274139 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.273832 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.281233 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.276747 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz"] Nov 01 21:08:17.281762 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.281729 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.281892 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.281812 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.281892 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.281842 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.282023 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.281895 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.282023 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.281921 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.282023 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.281968 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.282248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.282047 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjxv\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-kube-api-access-dtjxv\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.282248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.282161 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f591f6-2c37-47df-b609-08d801eb30fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.282248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.282197 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f591f6-2c37-47df-b609-08d801eb30fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.284313 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.284293 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.284606 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.284586 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw"] Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.388888 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f591f6-2c37-47df-b609-08d801eb30fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.388945 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.388982 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f591f6-2c37-47df-b609-08d801eb30fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389023 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389055 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389120 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389167 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389193 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwsp\" (UniqueName: \"kubernetes.io/projected/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-kube-api-access-cnwsp\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389224 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389262 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389298 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389335 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389364 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389401 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389446 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389470 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389500 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389537 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389577 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389638 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389663 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389691 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjxv\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-kube-api-access-dtjxv\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389720 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389754 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389787 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389813 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.389894 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcrrh\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-kube-api-access-zcrrh\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.392248 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f591f6-2c37-47df-b609-08d801eb30fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.392843 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.394394 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.396239 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.396503 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.397852 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f591f6-2c37-47df-b609-08d801eb30fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.398533 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.401050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.398547 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq"] Nov 01 21:08:17.404573 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.404543 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz"] Nov 01 21:08:17.488630 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.486715 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.488630 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.488435 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjxv\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-kube-api-access-dtjxv\") pod \"maas-default-gateway-openshift-default-58c6585447-pg8rw\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.491174 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491137 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.491343 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491187 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwsp\" (UniqueName: \"kubernetes.io/projected/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-kube-api-access-cnwsp\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.491343 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491229 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.491343 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491263 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.491343 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491305 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491343 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491373 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491406 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491445 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491501 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491525 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491555 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491582 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491616 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.491657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491642 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.492363 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491713 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcrrh\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-kube-api-access-zcrrh\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.492363 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491760 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.492363 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.491823 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.493247 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.493219 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.493425 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.493400 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.493835 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.493816 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.494214 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.494197 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.494330 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.494264 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.494566 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.494539 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.496248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.494565 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.496248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.494969 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.496248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.495132 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.496248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.495374 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:17.496248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.495703 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.497432 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.497375 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.499511 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.499487 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.499886 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.499867 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.500239 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.500218 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.589420 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.589374 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.589712 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.589687 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcrrh\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-kube-api-access-zcrrh\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.591128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.591104 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwsp\" (UniqueName: \"kubernetes.io/projected/c642c26c-c492-4b0b-9c1b-9a7cb89e7d97-kube-api-access-cnwsp\") pod \"maas-default-gateway-openshift-default-58c6585447-k7ndz\" (UID: \"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:17.591249 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.591233 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-m6rnq\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.595551 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.595532 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:17.604331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:17.604304 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:18.081659 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.081625 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw"] Nov 01 21:08:18.082294 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:08:18.082269 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f591f6_2c37_47df_b609_08d801eb30fa.slice/crio-1b019613c731655ae509f06aa203f0130aa194967de89af73040183729aa53ce WatchSource:0}: Error finding container 1b019613c731655ae509f06aa203f0130aa194967de89af73040183729aa53ce: Status 404 returned error can't find the container with id 1b019613c731655ae509f06aa203f0130aa194967de89af73040183729aa53ce Nov 01 21:08:18.085018 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.084982 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.085138 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.085053 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.085138 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.085103 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.100188 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.100165 4477 patch_prober.go:29] interesting pod/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 01 21:08:18.100284 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.100213 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 01 21:08:18.169749 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.167935 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq"] Nov 01 21:08:18.187931 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:08:18.187897 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f054e8_f0c0_4968_b8b2_c76fafb1876d.slice/crio-485a0be6eef08a695016bad5795ae0580f171589ab54097504d917ded9528304 WatchSource:0}: Error finding container 485a0be6eef08a695016bad5795ae0580f171589ab54097504d917ded9528304: Status 404 returned error can't find the container with id 485a0be6eef08a695016bad5795ae0580f171589ab54097504d917ded9528304 Nov 01 21:08:18.190299 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.190203 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz"] Nov 01 21:08:18.190728 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:08:18.190697 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc642c26c_c492_4b0b_9c1b_9a7cb89e7d97.slice/crio-6395b9c05b51fc92ab8c76db18ee5d3c29bf1257cf3dd08653677138b0e74e05 WatchSource:0}: Error finding container 6395b9c05b51fc92ab8c76db18ee5d3c29bf1257cf3dd08653677138b0e74e05: Status 404 returned error can't find the container with id 6395b9c05b51fc92ab8c76db18ee5d3c29bf1257cf3dd08653677138b0e74e05 Nov 01 21:08:18.191748 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.191710 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.191829 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.191774 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.191829 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.191807 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.193151 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.193118 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.193220 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.193193 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.193254 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.193231 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:18.290140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.278447 4477 patch_prober.go:29] interesting pod/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 21:08:18.290140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.278510 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 21:08:18.696104 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.692681 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" event={"ID":"19f054e8-f0c0-4968-b8b2-c76fafb1876d","Type":"ContainerStarted","Data":"26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f"} Nov 01 21:08:18.696104 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.692753 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" event={"ID":"19f054e8-f0c0-4968-b8b2-c76fafb1876d","Type":"ContainerStarted","Data":"485a0be6eef08a695016bad5795ae0580f171589ab54097504d917ded9528304"} Nov 01 21:08:18.707090 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.706257 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" event={"ID":"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97","Type":"ContainerStarted","Data":"0862f3b6a98fcb216c2357fce979b16a20521fb21dfe92eec323b6404f1de163"} Nov 01 21:08:18.707090 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.706325 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" event={"ID":"c642c26c-c492-4b0b-9c1b-9a7cb89e7d97","Type":"ContainerStarted","Data":"6395b9c05b51fc92ab8c76db18ee5d3c29bf1257cf3dd08653677138b0e74e05"} Nov 01 21:08:18.744100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.740387 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" event={"ID":"22f591f6-2c37-47df-b609-08d801eb30fa","Type":"ContainerStarted","Data":"f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a"} Nov 01 21:08:18.748099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.744373 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" event={"ID":"22f591f6-2c37-47df-b609-08d801eb30fa","Type":"ContainerStarted","Data":"1b019613c731655ae509f06aa203f0130aa194967de89af73040183729aa53ce"} Nov 01 21:08:18.841102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.840573 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podStartSLOduration=1.84055377 podStartE2EDuration="1.84055377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:18.812470949 +0000 UTC m=+2252.462522520" watchObservedRunningTime="2026-11-01 21:08:18.84055377 +0000 UTC m=+2252.490605316" Nov 01 21:08:18.988162 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:18.978312 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podStartSLOduration=1.978287059 podStartE2EDuration="1.978287059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:18.949283716 +0000 UTC m=+2252.599335287" watchObservedRunningTime="2026-11-01 21:08:18.978287059 +0000 UTC m=+2252.628338606" Nov 01 21:08:19.099097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.098053 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podStartSLOduration=2.09803028 podStartE2EDuration="2.09803028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:19.07669359 +0000 UTC m=+2252.726745136" watchObservedRunningTime="2026-11-01 21:08:19.09803028 +0000 UTC m=+2252.748081825" Nov 01 21:08:19.109120 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.105527 4477 patch_prober.go:29] interesting pod/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 01 21:08:19.109120 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.105588 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 01 21:08:19.271737 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.269546 4477 patch_prober.go:29] interesting pod/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 21:08:19.271737 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.269607 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 21:08:19.498876 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.498833 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:19.500456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.500065 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pg8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 21:08:19.500456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.500153 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 21:08:19.597100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.596220 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:19.597908 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.597873 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:19.598062 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.597958 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:19.605159 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.605118 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:19.606214 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.606185 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:19.606375 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:19.606234 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:20.111205 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.110354 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:08:20.111730 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.111712 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:08:20.308063 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.308032 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:08:20.309355 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.309334 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:08:20.311661 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.311636 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:08:20.509964 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.509874 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pg8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 21:08:20.510598 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.510564 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 21:08:20.688162 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.616617 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:20.688812 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.688767 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:20.713875 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.672756 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:20.713875 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:20.699215 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:21.502101 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:21.498997 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pg8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 21:08:21.502101 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:21.499051 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 21:08:21.602100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:21.600830 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:21.602100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:21.600918 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:21.627102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:21.608065 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:21.627102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:21.608178 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:22.505100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:22.501705 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pg8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 21:08:22.505100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:22.501763 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 21:08:22.617945 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:22.617891 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:22.618859 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:22.618828 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:22.619161 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:22.619141 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:22.619322 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:22.619301 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:23.499610 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:23.499559 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pg8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 21:08:23.499841 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:23.499635 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 21:08:23.596555 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:23.596516 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:23.596844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:23.596816 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:23.605497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:23.605463 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:23.605755 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:23.605731 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:24.507098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:24.504439 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pg8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 21:08:24.507098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:24.504500 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 21:08:24.600113 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:24.596493 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:24.600113 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:24.596554 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:24.608111 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:24.606430 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:24.608111 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:24.606501 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:25.501102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:25.500844 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pg8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 21:08:25.501102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:25.500917 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 21:08:25.596524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:25.596482 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:25.602103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:25.597803 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:25.606759 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:25.606728 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:25.607051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:25.607006 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:26.498015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:26.497976 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-pg8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 21:08:26.498318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:26.498280 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 21:08:26.595923 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:26.595887 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:26.596151 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:26.595943 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:26.605496 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:26.605460 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:26.605890 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:26.605524 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:27.596342 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:27.596303 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-m6rnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 01 21:08:27.596572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:27.596367 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 01 21:08:27.605441 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:27.605409 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-k7ndz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 01 21:08:27.605753 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:27.605704 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" podUID="c642c26c-c492-4b0b-9c1b-9a7cb89e7d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 01 21:08:28.441360 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:28.441326 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:28.444194 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:28.444173 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:28.445649 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:28.445631 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:08:28.830427 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:28.830378 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:28.830948 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:28.830851 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:28.831711 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:28.831693 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:08:29.079426 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:29.079390 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:29.080627 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:29.080560 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:29.081996 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:29.081972 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-k7ndz" Nov 01 21:08:36.553748 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:08:36.553701 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:08:43.009484 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.009452 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz"] Nov 01 21:08:43.013436 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.013412 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.065333 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.065294 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7091fd-openshift-default-dockercfg-pms7j\"" Nov 01 21:08:43.124634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.124579 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz"] Nov 01 21:08:43.147709 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.147672 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb621343-83ef-4b87-8c8e-e26e088560e5-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.156271 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.156226 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.156564 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.156542 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.163230 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.163191 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.168368 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.163492 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.168762 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.168737 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.169051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.169019 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mckbq\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-kube-api-access-mckbq\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.169556 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.169245 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.169740 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.169718 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271258 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271154 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mckbq\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-kube-api-access-mckbq\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271258 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271221 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271258 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271250 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271320 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb621343-83ef-4b87-8c8e-e26e088560e5-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271370 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271416 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271462 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271511 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271601 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271539 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271996 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271834 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.271996 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.271889 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.272165 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.272028 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.272165 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.272106 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.272252 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.272190 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb621343-83ef-4b87-8c8e-e26e088560e5-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.274102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.274063 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.274228 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.274212 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.332045 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.331980 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.343184 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.343063 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mckbq\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-kube-api-access-mckbq\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:43.630621 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:43.630573 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:44.088454 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:44.088415 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz"] Nov 01 21:08:44.123351 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:08:44.123315 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb621343_83ef_4b87_8c8e_e26e088560e5.slice/crio-744e800c34358eae7d7fa9909cb1626b47d67e89761953bb4c9f58e9bbe2fb0a WatchSource:0}: Error finding container 744e800c34358eae7d7fa9909cb1626b47d67e89761953bb4c9f58e9bbe2fb0a: Status 404 returned error can't find the container with id 744e800c34358eae7d7fa9909cb1626b47d67e89761953bb4c9f58e9bbe2fb0a Nov 01 21:08:44.127100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:44.127040 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:44.127221 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:44.127155 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:44.127221 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:44.127215 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:08:45.041828 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:45.041790 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" event={"ID":"fb621343-83ef-4b87-8c8e-e26e088560e5","Type":"ContainerStarted","Data":"2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7"} Nov 01 21:08:45.042025 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:45.041864 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" event={"ID":"fb621343-83ef-4b87-8c8e-e26e088560e5","Type":"ContainerStarted","Data":"744e800c34358eae7d7fa9909cb1626b47d67e89761953bb4c9f58e9bbe2fb0a"} Nov 01 21:08:45.207682 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:45.207618 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podStartSLOduration=3.20760366 podStartE2EDuration="3.20760366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:45.206422491 +0000 UTC m=+2278.856474036" watchObservedRunningTime="2026-11-01 21:08:45.20760366 +0000 UTC m=+2278.857655205" Nov 01 21:08:45.631449 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:45.631415 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:45.639056 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:45.639014 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:46.049378 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:46.049284 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:46.050557 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:46.050533 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:08:51.524289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:08:51.524235 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:09:06.556553 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:09:06.556459 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:09:20.283934 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.283890 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh"] Nov 01 21:09:20.288977 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.288951 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.341370 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.341335 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-044440-openshift-default-dockercfg-jwrzk\"" Nov 01 21:09:20.394621 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.394583 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh"] Nov 01 21:09:20.394864 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.394832 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-credential-socket\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.394984 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.394967 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clz2h\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-kube-api-access-clz2h\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.395167 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.395148 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-data\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.395253 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.395210 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istiod-ca-cert\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.395302 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.395278 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-podinfo\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.395340 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.395317 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-socket\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.395383 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.395341 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-certs\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.395534 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.395516 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-envoy\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.395580 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.395566 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-token\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.497548 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.497510 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-data\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.497763 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.497586 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istiod-ca-cert\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.497763 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.497616 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-podinfo\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498000 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.497971 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-data\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498198 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.497651 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-socket\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498293 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.498229 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-certs\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498359 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.498323 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-envoy\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498424 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.498371 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-token\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498424 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.498375 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-socket\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498424 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.498414 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istiod-ca-cert\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498593 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.498430 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-credential-socket\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.498593 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.498511 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clz2h\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-kube-api-access-clz2h\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.499143 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.499100 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-credential-socket\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.499248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.499231 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-certs\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.501386 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.501349 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-podinfo\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.501811 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.501789 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-envoy\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.557033 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.556918 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-token\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.559694 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.559642 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clz2h\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-kube-api-access-clz2h\") pod \"e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:20.603805 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:20.603756 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:21.046719 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:21.046691 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh"] Nov 01 21:09:21.048358 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:09:21.048329 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcceb2a08_115d_42b7_ad54_5bf4bbdbbabb.slice/crio-3c9dc3f4d91cce55184978b21121f1de3f5dce3ee5fb4f12930743d031cba164 WatchSource:0}: Error finding container 3c9dc3f4d91cce55184978b21121f1de3f5dce3ee5fb4f12930743d031cba164: Status 404 returned error can't find the container with id 3c9dc3f4d91cce55184978b21121f1de3f5dce3ee5fb4f12930743d031cba164 Nov 01 21:09:21.051042 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:21.051005 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:09:21.051166 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:21.051096 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:09:21.051166 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:21.051131 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:09:21.348066 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:21.348027 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" event={"ID":"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb","Type":"ContainerStarted","Data":"d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b"} Nov 01 21:09:21.348733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:21.348705 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" event={"ID":"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb","Type":"ContainerStarted","Data":"3c9dc3f4d91cce55184978b21121f1de3f5dce3ee5fb4f12930743d031cba164"} Nov 01 21:09:21.467429 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:21.467367 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" podStartSLOduration=1.467353093 podStartE2EDuration="1.467353093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:21.462987799 +0000 UTC m=+2315.113039347" watchObservedRunningTime="2026-11-01 21:09:21.467353093 +0000 UTC m=+2315.117404637" Nov 01 21:09:21.604250 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:21.604170 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:22.610099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:22.610054 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:23.362874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:23.362842 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:23.363854 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:23.363836 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:09:29.477806 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.477770 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs"] Nov 01 21:09:29.481906 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.481885 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.539317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.539284 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.539466 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.539443 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.539587 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.539562 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzwp\" (UniqueName: \"kubernetes.io/projected/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-kube-api-access-mnzwp\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.539665 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.539650 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-metrics-certs\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.583599 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.583573 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a41242-serving-cert\"" Nov 01 21:09:29.587049 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.587026 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs"] Nov 01 21:09:29.641245 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.641212 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-metrics-certs\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.641693 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.641656 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.641827 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.641810 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.641931 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.641916 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzwp\" (UniqueName: \"kubernetes.io/projected/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-kube-api-access-mnzwp\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.642043 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:09:29.642018 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-serving-cert: secret "maas-api-e2e-api-b-a41242-serving-cert" not found Nov 01 21:09:29.642309 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:09:29.642295 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls podName:8578e043-d4a0-4968-a7f7-b9fa052c8fb1 nodeName:}" failed. No retries permitted until 2026-11-01 21:09:30.142271513 +0000 UTC m=+2323.792323063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls") pod "maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" (UID: "8578e043-d4a0-4968-a7f7-b9fa052c8fb1") : secret "maas-api-e2e-api-b-a41242-serving-cert" not found Nov 01 21:09:29.642721 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.642683 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.644318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.644293 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-metrics-certs\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:29.699013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:29.698981 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzwp\" (UniqueName: \"kubernetes.io/projected/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-kube-api-access-mnzwp\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:30.152677 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:30.149669 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:30.155935 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:30.155904 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls\") pod \"maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:30.397340 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:30.397297 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:30.796974 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:30.796927 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs"] Nov 01 21:09:31.425253 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:31.425214 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" event={"ID":"8578e043-d4a0-4968-a7f7-b9fa052c8fb1","Type":"ContainerStarted","Data":"b8fa77ee391789a5a2c65d0ccd49c426af21a3e1f18e70fe2e02709f7cb51256"} Nov 01 21:09:32.434778 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:32.434739 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" event={"ID":"8578e043-d4a0-4968-a7f7-b9fa052c8fb1","Type":"ContainerStarted","Data":"ade76562919d18e6d08213dbd11cb8e8e9c10f4c226549c537225944e6bb87b3"} Nov 01 21:09:32.435339 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:32.434828 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:09:32.548617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:32.548561 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" podStartSLOduration=2.569318434 podStartE2EDuration="3.548541706s" totalImagesPullingTime="979.223272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:32.547355243 +0000 UTC m=+2326.197406806" watchObservedRunningTime="2026-11-01 21:09:32.548541706 +0000 UTC m=+2326.198593255" Nov 01 21:09:36.627101 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:09:36.624325 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:09:38.447394 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:09:38.447367 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:10:06.555024 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:10:06.554978 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:10:09.140065 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:09.140029 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:10:36.556532 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:10:36.556453 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:10:37.247449 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.247410 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz"] Nov 01 21:10:37.252608 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.252586 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.306621 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.306591 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz"] Nov 01 21:10:37.351991 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.351964 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-670df9-serving-cert\"" Nov 01 21:10:37.379196 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.379167 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2cb6\" (UniqueName: \"kubernetes.io/projected/fcb2250c-fc1b-4982-82d5-678fb31f35f3-kube-api-access-r2cb6\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.379360 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.379335 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb2250c-fc1b-4982-82d5-678fb31f35f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.379516 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.379491 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-metrics-certs\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.379573 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.379551 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.481239 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.481203 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb2250c-fc1b-4982-82d5-678fb31f35f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.481441 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.481276 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-metrics-certs\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.481441 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.481305 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.481441 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.481368 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2cb6\" (UniqueName: \"kubernetes.io/projected/fcb2250c-fc1b-4982-82d5-678fb31f35f3-kube-api-access-r2cb6\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.481621 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:10:37.481455 4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-serving-cert: secret "maas-api-e2e-api-b-670df9-serving-cert" not found Nov 01 21:10:37.481621 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:10:37.481546 4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls podName:fcb2250c-fc1b-4982-82d5-678fb31f35f3 nodeName:}" failed. No retries permitted until 2026-11-01 21:10:37.981522229 +0000 UTC m=+2391.631573767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls") pod "maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" (UID: "fcb2250c-fc1b-4982-82d5-678fb31f35f3") : secret "maas-api-e2e-api-b-670df9-serving-cert" not found Nov 01 21:10:37.481954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.481921 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb2250c-fc1b-4982-82d5-678fb31f35f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.483896 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.483876 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-metrics-certs\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.539891 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.539819 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2cb6\" (UniqueName: \"kubernetes.io/projected/fcb2250c-fc1b-4982-82d5-678fb31f35f3-kube-api-access-r2cb6\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.988217 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.988172 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:37.990809 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:37.990788 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls\") pod \"maas-api-e2e-api-b-670df9-7b5c876444-dlsnz\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:38.169913 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:38.169874 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:38.555710 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:38.555679 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz"] Nov 01 21:10:38.557240 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:10:38.557208 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb2250c_fc1b_4982_82d5_678fb31f35f3.slice/crio-e8d0355a1aab53131399a7655b865f59a96b7a192edb0f3ccdb2afb5fe593bd7 WatchSource:0}: Error finding container e8d0355a1aab53131399a7655b865f59a96b7a192edb0f3ccdb2afb5fe593bd7: Status 404 returned error can't find the container with id e8d0355a1aab53131399a7655b865f59a96b7a192edb0f3ccdb2afb5fe593bd7 Nov 01 21:10:38.921234 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:38.921185 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" event={"ID":"fcb2250c-fc1b-4982-82d5-678fb31f35f3","Type":"ContainerStarted","Data":"e8d0355a1aab53131399a7655b865f59a96b7a192edb0f3ccdb2afb5fe593bd7"} Nov 01 21:10:39.931163 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:39.931121 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" event={"ID":"fcb2250c-fc1b-4982-82d5-678fb31f35f3","Type":"ContainerStarted","Data":"7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0"} Nov 01 21:10:39.931579 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:39.931193 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:10:40.048602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:40.048534 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" podStartSLOduration=2.219784493 podStartE2EDuration="3.048516097s" totalImagesPullingTime="828.731604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:10:40.044959973 +0000 UTC m=+2393.695011563" watchObservedRunningTime="2026-11-01 21:10:40.048516097 +0000 UTC m=+2393.698567642" Nov 01 21:10:45.065673 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:45.065635 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:10:45.945420 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:10:45.945393 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:11:06.555466 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:11:06.555421 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:11:18.221928 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:18.221889 4477 generic.go:356] "Generic (PLEG): container finished" podID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerID="d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b" exitCode=0 Nov 01 21:11:18.222527 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:18.221964 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" event={"ID":"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb","Type":"ContainerDied","Data":"d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b"} Nov 01 21:11:18.222527 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:18.222486 4477 scope.go:122] "RemoveContainer" containerID="d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b" Nov 01 21:11:18.224333 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:18.224296 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:11:18.224444 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:18.224364 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:11:18.224444 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:18.224398 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:11:19.247826 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:19.247786 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" event={"ID":"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb","Type":"ContainerStarted","Data":"67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4"} Nov 01 21:11:19.604912 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:19.604879 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:11:19.606315 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:19.606279 4477 patch_prober.go:29] interesting pod/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 21:11:19.606454 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:19.606339 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 21:11:20.605194 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:20.605156 4477 patch_prober.go:29] interesting pod/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 21:11:20.605615 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:20.605221 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 21:11:21.605144 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:21.605066 4477 patch_prober.go:29] interesting pod/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 21:11:21.605374 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:21.605167 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 21:11:22.604908 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:22.604865 4477 patch_prober.go:29] interesting pod/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 21:11:22.605136 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:22.604938 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 21:11:23.368104 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:23.363450 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:11:23.605057 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:23.605022 4477 patch_prober.go:29] interesting pod/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 21:11:23.605287 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:23.605116 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 21:11:24.610362 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:24.610333 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:11:24.611479 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:24.611461 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:11:36.556720 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:11:36.556680 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:11:41.093736 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:41.093697 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:11:47.053481 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:47.053436 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4"] Nov 01 21:11:47.054042 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:47.053666 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" containerID="cri-o://1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe" gracePeriod=30 Nov 01 21:11:47.718634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:47.718597 4477 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:11:47.719225 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:47.719186 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:11:49.462824 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:49.462780 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj"] Nov 01 21:11:49.463335 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:49.463048 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" containerID="cri-o://52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b" gracePeriod=30 Nov 01 21:11:52.380031 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.380008 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 21:11:52.496342 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.496309 4477 generic.go:356] "Generic (PLEG): container finished" podID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerID="1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe" exitCode=0 Nov 01 21:11:52.496520 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.496393 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" Nov 01 21:11:52.505750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.505723 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" event={"ID":"e6667ff8-33fa-4bad-956a-1c9ae709b0bc","Type":"ContainerDied","Data":"1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe"} Nov 01 21:11:52.505893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.505772 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4" event={"ID":"e6667ff8-33fa-4bad-956a-1c9ae709b0bc","Type":"ContainerDied","Data":"da072ba6dc402445e4f88370518e119979ed5e9f61b2c8cabaf57eebbf28d47b"} Nov 01 21:11:52.505893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.505795 4477 scope.go:122] "RemoveContainer" containerID="1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe" Nov 01 21:11:52.545980 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.545959 4477 scope.go:122] "RemoveContainer" containerID="1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe" Nov 01 21:11:52.546417 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:11:52.546391 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe\": container with ID starting with 1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe not found: ID does not exist" containerID="1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe" Nov 01 21:11:52.546519 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.546430 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe"} err="failed to get container status \"1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe\": rpc error: code = NotFound desc = could not find container \"1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe\": container with ID starting with 1d8ce2bda3b7f2d7142e67a72fc4d736bdeb0ae12a7e2e5a974f635ce2996bfe not found: ID does not exist" Nov 01 21:11:52.555890 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.555844 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istiod-ca-cert\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556017 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.555895 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-envoy\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556017 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.555946 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-certs\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556017 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.555984 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-kube-api-access-b9b5m\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-kube-api-access-b9b5m\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556180 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.556018 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-token\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-token\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556180 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.556140 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-data\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556252 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.556243 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-credential-socket\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556373 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.556351 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-podinfo\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556460 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.556426 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-socket\") pod \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\" (UID: \"e6667ff8-33fa-4bad-956a-1c9ae709b0bc\") " Nov 01 21:11:52.556572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.556550 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-certs" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:52.556694 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.556673 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istiod-ca-cert" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:52.556996 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.556974 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.557115 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.557001 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.557348 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.557155 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-credential-socket" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:52.557464 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.557444 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-data" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:52.557538 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.557513 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-socket" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:52.559401 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.559366 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-envoy" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:52.560640 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.560616 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-kube-api-access-b9b5m" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "kube-api-access-b9b5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:52.560749 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.560720 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-token" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:52.563411 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.563389 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-podinfo" pod "e6667ff8-33fa-4bad-956a-1c9ae709b0bc" (UID: "e6667ff8-33fa-4bad-956a-1c9ae709b0bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:11:52.658251 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.658157 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.658251 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.658195 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.658251 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.658211 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.658251 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.658227 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9b5m\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-kube-api-access-b9b5m\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.658251 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.658244 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.658579 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.658258 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.658579 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.658273 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6667ff8-33fa-4bad-956a-1c9ae709b0bc-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.941646 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.941529 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4"] Nov 01 21:11:52.944775 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:52.944747 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-lxjj4"] Nov 01 21:11:54.504685 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.504654 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" path="/var/lib/kubelet/pods/e6667ff8-33fa-4bad-956a-1c9ae709b0bc/volumes" Nov 01 21:11:54.790622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.790597 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 21:11:54.884975 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.884944 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-istio-token\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-istio-token\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885190 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885165 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-data\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885238 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885200 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-kube-api-access-md8m7\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-kube-api-access-md8m7\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885240 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-certs\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885277 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926f1d5c-a975-439a-b8b3-b642316d7882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926f1d5c-a975-439a-b8b3-b642316d7882-istiod-ca-cert\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885384 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885299 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-socket\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885384 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885332 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/926f1d5c-a975-439a-b8b3-b642316d7882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926f1d5c-a975-439a-b8b3-b642316d7882-istio-podinfo\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885384 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885357 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-envoy\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885542 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885386 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-credential-socket\") pod \"926f1d5c-a975-439a-b8b3-b642316d7882\" (UID: \"926f1d5c-a975-439a-b8b3-b642316d7882\") " Nov 01 21:11:54.885542 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885476 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-data" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:54.885661 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885543 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-certs" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:54.885661 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885616 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926f1d5c-a975-439a-b8b3-b642316d7882-istiod-ca-cert" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:54.885772 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885701 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-socket" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:54.885833 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885773 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-credential-socket" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:54.886051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885957 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:54.886051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885984 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:54.886051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.885998 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:54.886051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.886012 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926f1d5c-a975-439a-b8b3-b642316d7882-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:54.886051 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.886025 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:54.887785 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.887763 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-kube-api-access-md8m7" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "kube-api-access-md8m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:54.887909 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.887793 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-istio-token" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:54.887972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.887902 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/926f1d5c-a975-439a-b8b3-b642316d7882-istio-podinfo" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:11:54.888112 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.888067 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-envoy" pod "926f1d5c-a975-439a-b8b3-b642316d7882" (UID: "926f1d5c-a975-439a-b8b3-b642316d7882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:11:54.987062 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.987027 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md8m7\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-kube-api-access-md8m7\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:54.987062 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.987058 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926f1d5c-a975-439a-b8b3-b642316d7882-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:54.987283 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.987069 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926f1d5c-a975-439a-b8b3-b642316d7882-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:54.987283 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:54.987107 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926f1d5c-a975-439a-b8b3-b642316d7882-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:55.525099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.523288 4477 generic.go:356] "Generic (PLEG): container finished" podID="926f1d5c-a975-439a-b8b3-b642316d7882" containerID="52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b" exitCode=0 Nov 01 21:11:55.525099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.523492 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" event={"ID":"926f1d5c-a975-439a-b8b3-b642316d7882","Type":"ContainerDied","Data":"52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b"} Nov 01 21:11:55.525099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.523547 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" event={"ID":"926f1d5c-a975-439a-b8b3-b642316d7882","Type":"ContainerDied","Data":"8544e44b7136b344295b4164ec664a8abf013c0aea42e16dd8659f508c77dad2"} Nov 01 21:11:55.525099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.523580 4477 scope.go:122] "RemoveContainer" containerID="52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b" Nov 01 21:11:55.525099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.523726 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj" Nov 01 21:11:55.562759 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.560955 4477 scope.go:122] "RemoveContainer" containerID="52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b" Nov 01 21:11:55.563299 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:11:55.563274 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b\": container with ID starting with 52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b not found: ID does not exist" containerID="52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b" Nov 01 21:11:55.563437 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.563312 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b"} err="failed to get container status \"52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b\": rpc error: code = NotFound desc = could not find container \"52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b\": container with ID starting with 52517ac3cb59aee76c54c7a519cee6bab92a97198dd6f60dd108fe9c320eb84b not found: ID does not exist" Nov 01 21:11:55.754285 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.754248 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj"] Nov 01 21:11:55.801664 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:55.801589 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lgsnj"] Nov 01 21:11:56.505279 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:11:56.505252 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" path="/var/lib/kubelet/pods/926f1d5c-a975-439a-b8b3-b642316d7882/volumes" Nov 01 21:12:01.050951 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:01.050915 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 01 21:12:01.051430 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:01.050980 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 01 21:12:01.589744 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:01.589707 4477 generic.go:356] "Generic (PLEG): container finished" podID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerID="2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7" exitCode=0 Nov 01 21:12:01.589939 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:01.589776 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" event={"ID":"fb621343-83ef-4b87-8c8e-e26e088560e5","Type":"ContainerDied","Data":"2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7"} Nov 01 21:12:01.590243 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:01.590229 4477 scope.go:122] "RemoveContainer" containerID="2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7" Nov 01 21:12:01.592038 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:01.592007 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:12:01.592140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:01.592120 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:12:01.592184 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:01.592164 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:12:02.606679 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:02.606637 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" event={"ID":"fb621343-83ef-4b87-8c8e-e26e088560e5","Type":"ContainerStarted","Data":"8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1"} Nov 01 21:12:02.638326 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:02.638278 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:12:02.638559 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:02.638526 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 01 21:12:02.638636 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:02.638569 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 01 21:12:03.632518 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:03.632476 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 01 21:12:03.633271 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:03.633222 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 01 21:12:04.631506 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:04.631465 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 01 21:12:04.631836 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:04.631801 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 01 21:12:05.638729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:05.638677 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 01 21:12:05.639283 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:05.638746 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 01 21:12:06.585776 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:12:06.585738 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:12:06.631722 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:06.631689 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 01 21:12:06.632085 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:06.632040 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 01 21:12:06.648122 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:06.639631 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv"] Nov 01 21:12:06.648122 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:06.639888 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" containerID="cri-o://d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11" gracePeriod=30 Nov 01 21:12:07.650619 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:07.650581 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 01 21:12:07.651256 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:07.651215 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 01 21:12:08.635761 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:08.635722 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:12:08.636017 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:08.635997 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:12:08.636726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:08.636706 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:12:11.927726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:11.927668 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 21:12:12.081214 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081178 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adba17bf-847c-4f95-a0d6-34c71729277e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adba17bf-847c-4f95-a0d6-34c71729277e-istiod-ca-cert\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.081415 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081227 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-certs\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.081415 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081280 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-data\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.081415 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081395 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-credential-socket\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.081602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081432 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-kube-api-access-vw9fq\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-kube-api-access-vw9fq\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.081602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081480 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-istio-token\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-istio-token\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.081602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081533 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-envoy\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.081602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081543 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-certs" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:12.081602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081596 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-socket\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.081869 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081663 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-data" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:12.081869 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081676 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-credential-socket" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:12.081869 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081692 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adba17bf-847c-4f95-a0d6-34c71729277e-istiod-ca-cert" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:12.081869 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081700 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adba17bf-847c-4f95-a0d6-34c71729277e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adba17bf-847c-4f95-a0d6-34c71729277e-istio-podinfo\") pod \"adba17bf-847c-4f95-a0d6-34c71729277e\" (UID: \"adba17bf-847c-4f95-a0d6-34c71729277e\") " Nov 01 21:12:12.082110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.081950 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-socket" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:12.082473 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.082439 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.082657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.082631 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adba17bf-847c-4f95-a0d6-34c71729277e-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.082657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.082658 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.082832 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.082672 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.082832 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.082686 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.084061 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.084041 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-istio-token" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:12.084333 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.084314 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adba17bf-847c-4f95-a0d6-34c71729277e-istio-podinfo" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:12:12.084485 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.084462 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-envoy" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:12.084574 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.084534 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-kube-api-access-vw9fq" pod "adba17bf-847c-4f95-a0d6-34c71729277e" (UID: "adba17bf-847c-4f95-a0d6-34c71729277e"). InnerVolumeSpecName "kube-api-access-vw9fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:12.184290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.184206 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9fq\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-kube-api-access-vw9fq\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.184290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.184234 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adba17bf-847c-4f95-a0d6-34c71729277e-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.184290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.184244 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adba17bf-847c-4f95-a0d6-34c71729277e-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.184290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.184254 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adba17bf-847c-4f95-a0d6-34c71729277e-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:12.719904 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.719873 4477 generic.go:356] "Generic (PLEG): container finished" podID="adba17bf-847c-4f95-a0d6-34c71729277e" containerID="d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11" exitCode=0 Nov 01 21:12:12.720113 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.719921 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" Nov 01 21:12:12.720113 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.719968 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" event={"ID":"adba17bf-847c-4f95-a0d6-34c71729277e","Type":"ContainerDied","Data":"d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11"} Nov 01 21:12:12.720113 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.720028 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv" event={"ID":"adba17bf-847c-4f95-a0d6-34c71729277e","Type":"ContainerDied","Data":"dca5b440efdb9fea8f21854c16d20891804b49d8098d2fed19281c0f70b13c78"} Nov 01 21:12:12.720113 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.720053 4477 scope.go:122] "RemoveContainer" containerID="d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11" Nov 01 21:12:12.730227 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.730200 4477 scope.go:122] "RemoveContainer" containerID="d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11" Nov 01 21:12:12.730511 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:12:12.730489 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11\": container with ID starting with d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11 not found: ID does not exist" containerID="d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11" Nov 01 21:12:12.730588 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.730523 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11"} err="failed to get container status \"d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11\": rpc error: code = NotFound desc = could not find container \"d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11\": container with ID starting with d13323c7858832caaf7a517e9d71118f819852321a708a87f28382f4030d0a11 not found: ID does not exist" Nov 01 21:12:12.936244 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.936209 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv"] Nov 01 21:12:12.968894 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:12.968860 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-dcbhv"] Nov 01 21:12:14.504317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:14.504288 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" path="/var/lib/kubelet/pods/adba17bf-847c-4f95-a0d6-34c71729277e/volumes" Nov 01 21:12:21.517646 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:21.517611 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:12:36.557025 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:12:36.556976 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:12:44.935955 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.935909 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp"] Nov 01 21:12:44.936505 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936490 4477 cpu_manager.go:414] "containerMap: removing container" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" Nov 01 21:12:44.936550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936506 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" Nov 01 21:12:44.936550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936516 4477 cpu_manager.go:414] "containerMap: removing container" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" Nov 01 21:12:44.936550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936537 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" Nov 01 21:12:44.936646 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936559 4477 cpu_manager.go:414] "containerMap: removing container" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" Nov 01 21:12:44.936646 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936564 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" Nov 01 21:12:44.936646 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936641 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="926f1d5c-a975-439a-b8b3-b642316d7882" containerName="istio-proxy" Nov 01 21:12:44.936741 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936650 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6667ff8-33fa-4bad-956a-1c9ae709b0bc" containerName="istio-proxy" Nov 01 21:12:44.936741 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.936659 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="adba17bf-847c-4f95-a0d6-34c71729277e" containerName="istio-proxy" Nov 01 21:12:44.940054 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.940036 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995294 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995261 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995294 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995309 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995577 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995384 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995577 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995420 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995577 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995493 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbk5l\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-kube-api-access-xbk5l\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995577 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995546 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995800 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995591 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995800 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995670 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:44.995800 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:44.995727 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.046489 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.046447 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp"] Nov 01 21:12:45.097376 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097345 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.097572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097399 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.097572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097431 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.097572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097460 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbk5l\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-kube-api-access-xbk5l\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.097572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097533 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.097572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097567 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.097841 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097622 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.097841 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097694 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.097841 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097766 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.098017 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.097869 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.098121 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.098015 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.098121 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.098102 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.098234 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.098158 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.098234 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.098202 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.100208 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.100184 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.100567 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.100544 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.154736 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.154704 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.154890 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.154779 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbk5l\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-kube-api-access-xbk5l\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.255117 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.254987 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:45.689543 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.689518 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp"] Nov 01 21:12:45.691246 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:12:45.691222 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1e1ac1_727a_48b3_b0e3_f199ba1f5d41.slice/crio-57461467916b43b8c8fda1bb0383b7f246081ec0f14353a7c42aa01bec56d61c WatchSource:0}: Error finding container 57461467916b43b8c8fda1bb0383b7f246081ec0f14353a7c42aa01bec56d61c: Status 404 returned error can't find the container with id 57461467916b43b8c8fda1bb0383b7f246081ec0f14353a7c42aa01bec56d61c Nov 01 21:12:45.693806 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.693774 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:12:45.693869 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.693839 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:12:45.693903 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.693874 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:12:45.967032 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.966902 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" event={"ID":"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41","Type":"ContainerStarted","Data":"542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c"} Nov 01 21:12:45.967546 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:45.967038 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" event={"ID":"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41","Type":"ContainerStarted","Data":"57461467916b43b8c8fda1bb0383b7f246081ec0f14353a7c42aa01bec56d61c"} Nov 01 21:12:46.086775 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:46.086716 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" podStartSLOduration=2.086701671 podStartE2EDuration="2.086701671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:12:46.083624319 +0000 UTC m=+2519.733675865" watchObservedRunningTime="2026-11-01 21:12:46.086701671 +0000 UTC m=+2519.736753216" Nov 01 21:12:46.255534 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:46.255440 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:46.257180 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:46.257154 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 21:12:46.257333 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:46.257210 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 21:12:47.255830 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:47.255786 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 21:12:47.256277 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:47.255848 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 21:12:48.255723 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:48.255677 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 21:12:48.255955 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:48.255746 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 21:12:49.255815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:49.255778 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 21:12:49.256011 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:49.255832 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 21:12:50.255665 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:50.255625 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 21:12:50.255976 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:50.255942 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 21:12:51.255683 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:51.255635 4477 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 21:12:51.256238 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:51.255696 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 21:12:52.260572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:52.260544 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:52.261027 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:52.261005 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:52.261835 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:52.261816 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:12:59.287168 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.287131 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt"] Nov 01 21:12:59.287666 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.287413 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" podUID="06488271-ae4a-4f4e-8d6f-955a029ca19a" containerName="maas-api" containerID="cri-o://524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7" gracePeriod=30 Nov 01 21:12:59.607401 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.607376 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:12:59.630185 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.630160 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06488271-ae4a-4f4e-8d6f-955a029ca19a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06488271-ae4a-4f4e-8d6f-955a029ca19a-openshift-service-ca\") pod \"06488271-ae4a-4f4e-8d6f-955a029ca19a\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " Nov 01 21:12:59.630361 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.630203 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-metrics-certs\") pod \"06488271-ae4a-4f4e-8d6f-955a029ca19a\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " Nov 01 21:12:59.630361 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.630342 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls\") pod \"06488271-ae4a-4f4e-8d6f-955a029ca19a\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " Nov 01 21:12:59.630495 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.630473 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06488271-ae4a-4f4e-8d6f-955a029ca19a-kube-api-access-4dbv2\" (UniqueName: \"kubernetes.io/projected/06488271-ae4a-4f4e-8d6f-955a029ca19a-kube-api-access-4dbv2\") pod \"06488271-ae4a-4f4e-8d6f-955a029ca19a\" (UID: \"06488271-ae4a-4f4e-8d6f-955a029ca19a\") " Nov 01 21:12:59.630550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.630519 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06488271-ae4a-4f4e-8d6f-955a029ca19a-openshift-service-ca" pod "06488271-ae4a-4f4e-8d6f-955a029ca19a" (UID: "06488271-ae4a-4f4e-8d6f-955a029ca19a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:59.630947 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.630925 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06488271-ae4a-4f4e-8d6f-955a029ca19a-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:59.632679 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.632653 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls" pod "06488271-ae4a-4f4e-8d6f-955a029ca19a" (UID: "06488271-ae4a-4f4e-8d6f-955a029ca19a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:12:59.633086 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.633053 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-metrics-certs" pod "06488271-ae4a-4f4e-8d6f-955a029ca19a" (UID: "06488271-ae4a-4f4e-8d6f-955a029ca19a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:12:59.633160 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.633114 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06488271-ae4a-4f4e-8d6f-955a029ca19a-kube-api-access-4dbv2" pod "06488271-ae4a-4f4e-8d6f-955a029ca19a" (UID: "06488271-ae4a-4f4e-8d6f-955a029ca19a"). InnerVolumeSpecName "kube-api-access-4dbv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:59.732210 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.732169 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:59.732210 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.732209 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06488271-ae4a-4f4e-8d6f-955a029ca19a-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:59.732494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:12:59.732225 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dbv2\" (UniqueName: \"kubernetes.io/projected/06488271-ae4a-4f4e-8d6f-955a029ca19a-kube-api-access-4dbv2\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:00.071924 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.071883 4477 generic.go:356] "Generic (PLEG): container finished" podID="06488271-ae4a-4f4e-8d6f-955a029ca19a" containerID="524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7" exitCode=0 Nov 01 21:13:00.072167 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.071940 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" Nov 01 21:13:00.072167 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.071966 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" event={"ID":"06488271-ae4a-4f4e-8d6f-955a029ca19a","Type":"ContainerDied","Data":"524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7"} Nov 01 21:13:00.072167 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.072022 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt" event={"ID":"06488271-ae4a-4f4e-8d6f-955a029ca19a","Type":"ContainerDied","Data":"60cc96e58c8b4507cd199019270d9043717e5b95e769d7d5d6ac2bb3d71adecd"} Nov 01 21:13:00.072167 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.072046 4477 scope.go:122] "RemoveContainer" containerID="524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7" Nov 01 21:13:00.083732 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.083711 4477 scope.go:122] "RemoveContainer" containerID="524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7" Nov 01 21:13:00.084038 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:13:00.084019 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7\": container with ID starting with 524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7 not found: ID does not exist" containerID="524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7" Nov 01 21:13:00.084134 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.084059 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7"} err="failed to get container status \"524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7\": rpc error: code = NotFound desc = could not find container \"524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7\": container with ID starting with 524701befaf489e7a3f59f96444e456a8d4e537e957051ec97db3544496640a7 not found: ID does not exist" Nov 01 21:13:00.236560 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.236522 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt"] Nov 01 21:13:00.241907 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.241879 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4ac1ef-68b44645dd-zlrbt"] Nov 01 21:13:00.506042 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:00.505956 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06488271-ae4a-4f4e-8d6f-955a029ca19a" path="/var/lib/kubelet/pods/06488271-ae4a-4f4e-8d6f-955a029ca19a/volumes" Nov 01 21:13:06.557033 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:13:06.556984 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:13:08.025064 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:08.025033 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4"] Nov 01 21:13:08.025620 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:08.025298 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" containerID="cri-o://5a016b859f02558689b53c951ce822343f09f0277e6c1ae12f064cb23fde4d3d" gracePeriod=30 Nov 01 21:13:08.099132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:08.090293 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf"] Nov 01 21:13:08.099132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:08.097567 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8"] Nov 01 21:13:08.142101 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:08.134308 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m"] Nov 01 21:13:13.178034 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.178002 4477 generic.go:356] "Generic (PLEG): container finished" podID="b6c13059-d000-41e5-90af-2536921d5165" containerID="5a016b859f02558689b53c951ce822343f09f0277e6c1ae12f064cb23fde4d3d" exitCode=0 Nov 01 21:13:13.178471 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.178179 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" event={"ID":"b6c13059-d000-41e5-90af-2536921d5165","Type":"ContainerDied","Data":"5a016b859f02558689b53c951ce822343f09f0277e6c1ae12f064cb23fde4d3d"} Nov 01 21:13:13.178471 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.178226 4477 scope.go:122] "RemoveContainer" containerID="2f7d0e6fc5fbbd5eaeb9982ddb2668bde088c5b33fc3978a3970555655db097f" Nov 01 21:13:13.308126 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.308047 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:13:13.413269 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413235 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-socket\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413492 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413297 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-credential-socket\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413492 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413346 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-envoy\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413492 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413424 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-kube-api-access-2svs7\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-kube-api-access-2svs7\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413521 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c13059-d000-41e5-90af-2536921d5165-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c13059-d000-41e5-90af-2536921d5165-istiod-ca-cert\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413566 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-certs\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413561 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-socket" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:13.413674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413606 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6c13059-d000-41e5-90af-2536921d5165-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c13059-d000-41e5-90af-2536921d5165-istio-podinfo\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413662 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-data\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413688 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-credential-socket" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:13.413954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413705 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-istio-token\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-istio-token\") pod \"b6c13059-d000-41e5-90af-2536921d5165\" (UID: \"b6c13059-d000-41e5-90af-2536921d5165\") " Nov 01 21:13:13.413954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.413926 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c13059-d000-41e5-90af-2536921d5165-istiod-ca-cert" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:13.414167 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.414136 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-data" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:13.414503 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.414419 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c13059-d000-41e5-90af-2536921d5165-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:13.414503 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.414444 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:13.414503 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.414467 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:13.414503 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.414481 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:13.414885 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.414628 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-certs" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:13.416253 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.416202 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-istio-token" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:13.416362 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.416313 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-kube-api-access-2svs7" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "kube-api-access-2svs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:13.416487 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.416468 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6c13059-d000-41e5-90af-2536921d5165-istio-podinfo" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:13:13.416739 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.416717 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-envoy" pod "b6c13059-d000-41e5-90af-2536921d5165" (UID: "b6c13059-d000-41e5-90af-2536921d5165"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:13.516110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.516029 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:13.516110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.516061 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:13.516110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.516101 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2svs7\" (UniqueName: \"kubernetes.io/projected/b6c13059-d000-41e5-90af-2536921d5165-kube-api-access-2svs7\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:13.516110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.516116 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c13059-d000-41e5-90af-2536921d5165-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:13.516393 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:13.516131 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c13059-d000-41e5-90af-2536921d5165-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:14.186470 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:14.186431 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" Nov 01 21:13:14.186470 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:14.186436 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4" event={"ID":"b6c13059-d000-41e5-90af-2536921d5165","Type":"ContainerDied","Data":"735c01731e8270e0cc48e3a7aa8856f76c533e725659028d6fd89bdeb6cb2ab9"} Nov 01 21:13:14.187055 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:14.186505 4477 scope.go:122] "RemoveContainer" containerID="5a016b859f02558689b53c951ce822343f09f0277e6c1ae12f064cb23fde4d3d" Nov 01 21:13:14.400556 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:14.400519 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4"] Nov 01 21:13:14.403640 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:14.403612 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4ac1ef-openshift-default-54d7ccf455-j89g4"] Nov 01 21:13:14.504600 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:14.504507 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c13059-d000-41e5-90af-2536921d5165" path="/var/lib/kubelet/pods/b6c13059-d000-41e5-90af-2536921d5165/volumes" Nov 01 21:13:20.479083 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:20.479021 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:13:23.133021 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.132937 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" containerID="cri-o://3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b" gracePeriod=45 Nov 01 21:13:23.167387 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.167326 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" containerID="cri-o://2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512" gracePeriod=45 Nov 01 21:13:23.184060 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.184032 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" containerID="cri-o://13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8" gracePeriod=45 Nov 01 21:13:23.202222 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.202189 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused" Nov 01 21:13:23.317479 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.311403 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused" Nov 01 21:13:23.491715 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.491691 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 21:13:23.497040 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.497016 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 21:13:23.510733 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.510708 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 21:13:23.564022 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.563987 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09423e7-1763-473d-b671-729f2d634515-tls-certs\" (UniqueName: \"kubernetes.io/secret/f09423e7-1763-473d-b671-729f2d634515-tls-certs\") pod \"f09423e7-1763-473d-b671-729f2d634515\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " Nov 01 21:13:23.564256 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.564092 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09423e7-1763-473d-b671-729f2d634515-kube-api-access-k4dpr\" (UniqueName: \"kubernetes.io/projected/f09423e7-1763-473d-b671-729f2d634515-kube-api-access-k4dpr\") pod \"f09423e7-1763-473d-b671-729f2d634515\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " Nov 01 21:13:23.564324 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.564290 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-dshm\") pod \"f09423e7-1763-473d-b671-729f2d634515\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " Nov 01 21:13:23.564628 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.564602 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-tmp-dir\") pod \"f09423e7-1763-473d-b671-729f2d634515\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " Nov 01 21:13:23.564756 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.564633 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-model-cache\") pod \"f09423e7-1763-473d-b671-729f2d634515\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " Nov 01 21:13:23.564756 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.564658 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-home\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-home\") pod \"f09423e7-1763-473d-b671-729f2d634515\" (UID: \"f09423e7-1763-473d-b671-729f2d634515\") " Nov 01 21:13:23.564913 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.564889 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-tmp-dir" pod "f09423e7-1763-473d-b671-729f2d634515" (UID: "f09423e7-1763-473d-b671-729f2d634515"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.564978 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.564920 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-model-cache" pod "f09423e7-1763-473d-b671-729f2d634515" (UID: "f09423e7-1763-473d-b671-729f2d634515"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.565271 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.565254 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.565343 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.565273 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.565741 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.565704 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-home" pod "f09423e7-1763-473d-b671-729f2d634515" (UID: "f09423e7-1763-473d-b671-729f2d634515"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.567012 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.566989 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-dshm" pod "f09423e7-1763-473d-b671-729f2d634515" (UID: "f09423e7-1763-473d-b671-729f2d634515"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.567099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.567027 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09423e7-1763-473d-b671-729f2d634515-kube-api-access-k4dpr" pod "f09423e7-1763-473d-b671-729f2d634515" (UID: "f09423e7-1763-473d-b671-729f2d634515"). InnerVolumeSpecName "kube-api-access-k4dpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:23.567606 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.567583 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09423e7-1763-473d-b671-729f2d634515-tls-certs" pod "f09423e7-1763-473d-b671-729f2d634515" (UID: "f09423e7-1763-473d-b671-729f2d634515"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:23.665855 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.665763 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tls-certs\") pod \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " Nov 01 21:13:23.665855 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.665809 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-dshm\") pod \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " Nov 01 21:13:23.666132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.665885 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tmp-dir\") pod \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " Nov 01 21:13:23.666132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.665914 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-dshm\") pod \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " Nov 01 21:13:23.666132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.665998 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tmp-dir\") pod \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " Nov 01 21:13:23.666132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.666030 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-kube-api-access-6gs4j\" (UniqueName: \"kubernetes.io/projected/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-kube-api-access-6gs4j\") pod \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " Nov 01 21:13:23.666132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.666087 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tls-certs\") pod \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " Nov 01 21:13:23.666132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.666124 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0288ca11-f321-4f10-b59f-8f1a4e7c293b-kube-api-access-6v2rm\" (UniqueName: \"kubernetes.io/projected/0288ca11-f321-4f10-b59f-8f1a4e7c293b-kube-api-access-6v2rm\") pod \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " Nov 01 21:13:23.666474 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.666175 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-home\") pod \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " Nov 01 21:13:23.666474 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.666250 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-home\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-home\") pod \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " Nov 01 21:13:23.666474 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.666295 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-model-cache\") pod \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\" (UID: \"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e\") " Nov 01 21:13:23.666474 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.666366 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-model-cache\") pod \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\" (UID: \"0288ca11-f321-4f10-b59f-8f1a4e7c293b\") " Nov 01 21:13:23.667207 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.667182 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.668155 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.667398 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tmp-dir" pod "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" (UID: "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.668155 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.667703 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-model-cache" pod "0288ca11-f321-4f10-b59f-8f1a4e7c293b" (UID: "0288ca11-f321-4f10-b59f-8f1a4e7c293b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.668155 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.667845 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tmp-dir" pod "0288ca11-f321-4f10-b59f-8f1a4e7c293b" (UID: "0288ca11-f321-4f10-b59f-8f1a4e7c293b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.668155 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.667856 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-model-cache" pod "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" (UID: "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.668155 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.667887 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-home" pod "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" (UID: "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.668155 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.668019 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-home" pod "0288ca11-f321-4f10-b59f-8f1a4e7c293b" (UID: "0288ca11-f321-4f10-b59f-8f1a4e7c293b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.668641 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.668626 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09423e7-1763-473d-b671-729f2d634515-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.668802 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.668778 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09423e7-1763-473d-b671-729f2d634515-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.668920 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.668805 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dpr\" (UniqueName: \"kubernetes.io/projected/f09423e7-1763-473d-b671-729f2d634515-kube-api-access-k4dpr\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.669452 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.669407 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tls-certs" pod "0288ca11-f321-4f10-b59f-8f1a4e7c293b" (UID: "0288ca11-f321-4f10-b59f-8f1a4e7c293b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:23.669736 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.669710 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-dshm" pod "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" (UID: "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.669890 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.669870 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0288ca11-f321-4f10-b59f-8f1a4e7c293b-kube-api-access-6v2rm" pod "0288ca11-f321-4f10-b59f-8f1a4e7c293b" (UID: "0288ca11-f321-4f10-b59f-8f1a4e7c293b"). InnerVolumeSpecName "kube-api-access-6v2rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:23.670336 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.670309 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-dshm" pod "0288ca11-f321-4f10-b59f-8f1a4e7c293b" (UID: "0288ca11-f321-4f10-b59f-8f1a4e7c293b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.670684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.670659 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tls-certs" pod "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" (UID: "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:23.672202 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.672176 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-kube-api-access-6gs4j" pod "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" (UID: "e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e"). InnerVolumeSpecName "kube-api-access-6gs4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:23.770689 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770652 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770689 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770690 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770705 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770718 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770731 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770746 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gs4j\" (UniqueName: \"kubernetes.io/projected/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-kube-api-access-6gs4j\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770758 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770772 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v2rm\" (UniqueName: \"kubernetes.io/projected/0288ca11-f321-4f10-b59f-8f1a4e7c293b-kube-api-access-6v2rm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770785 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770798 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770811 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.770910 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:23.770825 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0288ca11-f321-4f10-b59f-8f1a4e7c293b-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:24.256779 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.256750 4477 generic.go:356] "Generic (PLEG): container finished" podID="f09423e7-1763-473d-b671-729f2d634515" containerID="3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b" exitCode=0 Nov 01 21:13:24.257237 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.256795 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" Nov 01 21:13:24.257237 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.256849 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" event={"ID":"f09423e7-1763-473d-b671-729f2d634515","Type":"ContainerDied","Data":"3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b"} Nov 01 21:13:24.257237 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.256891 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8" event={"ID":"f09423e7-1763-473d-b671-729f2d634515","Type":"ContainerDied","Data":"de24b3cc0d322c07642cd10a06e61b74fb86dffa8a9c4e469c198701a2ad37d5"} Nov 01 21:13:24.257237 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.256913 4477 scope.go:122] "RemoveContainer" containerID="3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b" Nov 01 21:13:24.258392 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.258372 4477 generic.go:356] "Generic (PLEG): container finished" podID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerID="13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8" exitCode=0 Nov 01 21:13:24.258526 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.258426 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" Nov 01 21:13:24.258526 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.258468 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" event={"ID":"0288ca11-f321-4f10-b59f-8f1a4e7c293b","Type":"ContainerDied","Data":"13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8"} Nov 01 21:13:24.258526 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.258516 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m" event={"ID":"0288ca11-f321-4f10-b59f-8f1a4e7c293b","Type":"ContainerDied","Data":"adbecb667cd1c56a7fb3376b2adab4637c5638d3d57e363b50ac8d67184262c3"} Nov 01 21:13:24.260064 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.260039 4477 generic.go:356] "Generic (PLEG): container finished" podID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerID="2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512" exitCode=0 Nov 01 21:13:24.260208 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.260111 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" event={"ID":"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e","Type":"ContainerDied","Data":"2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512"} Nov 01 21:13:24.260208 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.260139 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" event={"ID":"e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e","Type":"ContainerDied","Data":"af1bf8e20fa97298b8388efb815e42704b0efb5f9c6e6e13d9b08134cbac0d5a"} Nov 01 21:13:24.260208 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.260195 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf" Nov 01 21:13:24.267592 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.267573 4477 scope.go:122] "RemoveContainer" containerID="3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b" Nov 01 21:13:24.268039 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:13:24.267897 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b\": container with ID starting with 3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b not found: ID does not exist" containerID="3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b" Nov 01 21:13:24.268039 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.267933 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b"} err="failed to get container status \"3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b\": rpc error: code = NotFound desc = could not find container \"3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b\": container with ID starting with 3c82b98b8dbbd1643a0170a40697b3f679922084cb7412642da10f455a0e6c6b not found: ID does not exist" Nov 01 21:13:24.268039 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.267957 4477 scope.go:122] "RemoveContainer" containerID="13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8" Nov 01 21:13:24.278956 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.278938 4477 scope.go:122] "RemoveContainer" containerID="13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8" Nov 01 21:13:24.279271 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:13:24.279251 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8\": container with ID starting with 13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8 not found: ID does not exist" containerID="13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8" Nov 01 21:13:24.279328 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.279280 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8"} err="failed to get container status \"13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8\": rpc error: code = NotFound desc = could not find container \"13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8\": container with ID starting with 13951e270297e16fcc5d6e43da1e35e4dcec67e8c9cd6e418d88a76b00e4acf8 not found: ID does not exist" Nov 01 21:13:24.279328 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.279296 4477 scope.go:122] "RemoveContainer" containerID="2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512" Nov 01 21:13:24.288124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.288102 4477 scope.go:122] "RemoveContainer" containerID="2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512" Nov 01 21:13:24.288411 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:13:24.288391 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512\": container with ID starting with 2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512 not found: ID does not exist" containerID="2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512" Nov 01 21:13:24.288468 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.288420 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512"} err="failed to get container status \"2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512\": rpc error: code = NotFound desc = could not find container \"2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512\": container with ID starting with 2cf73574e2bed351a1a73f73899418965f57b2fa4d02999f3dd18a1cfe12e512 not found: ID does not exist" Nov 01 21:13:24.420456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.420423 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8"] Nov 01 21:13:24.423361 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.423314 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-w2-cd248f-kserve-68cbd7474d-jgbv8"] Nov 01 21:13:24.505894 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.505866 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09423e7-1763-473d-b671-729f2d634515" path="/var/lib/kubelet/pods/f09423e7-1763-473d-b671-729f2d634515/volumes" Nov 01 21:13:24.530662 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.530626 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m"] Nov 01 21:13:24.533574 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.533538 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/simulator-w2-cd248f-kserve-5ccc68fb8d-4n69m"] Nov 01 21:13:24.640221 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.640143 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf"] Nov 01 21:13:24.643316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:24.643290 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/distinct-2-w2-cd248f-kserve-5777bcbdb7-c4zkf"] Nov 01 21:13:26.506445 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:26.506412 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" path="/var/lib/kubelet/pods/0288ca11-f321-4f10-b59f-8f1a4e7c293b/volumes" Nov 01 21:13:26.506857 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:26.506843 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" path="/var/lib/kubelet/pods/e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e/volumes" Nov 01 21:13:32.168217 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.168174 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8"] Nov 01 21:13:32.168693 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.168458 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" containerID="cri-o://3efd509a119221d7c0f4d1a7d75da516102265711ed2e05109ce9acc94ab70a2" gracePeriod=30 Nov 01 21:13:32.268806 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.268766 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj"] Nov 01 21:13:32.269088 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.269028 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" podUID="2416d0e0-25b6-485a-8612-723c279a3e21" containerName="maas-api" containerID="cri-o://a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968" gracePeriod=30 Nov 01 21:13:32.704440 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.704356 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:13:32.714399 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.714375 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2416d0e0-25b6-485a-8612-723c279a3e21-kube-api-access-xwxxb\" (UniqueName: \"kubernetes.io/projected/2416d0e0-25b6-485a-8612-723c279a3e21-kube-api-access-xwxxb\") pod \"2416d0e0-25b6-485a-8612-723c279a3e21\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " Nov 01 21:13:32.714545 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.714440 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2416d0e0-25b6-485a-8612-723c279a3e21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416d0e0-25b6-485a-8612-723c279a3e21-openshift-service-ca\") pod \"2416d0e0-25b6-485a-8612-723c279a3e21\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " Nov 01 21:13:32.714615 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.714539 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls\") pod \"2416d0e0-25b6-485a-8612-723c279a3e21\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " Nov 01 21:13:32.714740 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.714719 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-metrics-certs\") pod \"2416d0e0-25b6-485a-8612-723c279a3e21\" (UID: \"2416d0e0-25b6-485a-8612-723c279a3e21\") " Nov 01 21:13:32.714864 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.714842 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2416d0e0-25b6-485a-8612-723c279a3e21-openshift-service-ca" pod "2416d0e0-25b6-485a-8612-723c279a3e21" (UID: "2416d0e0-25b6-485a-8612-723c279a3e21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:32.715362 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.715293 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416d0e0-25b6-485a-8612-723c279a3e21-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:32.716945 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.716914 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls" pod "2416d0e0-25b6-485a-8612-723c279a3e21" (UID: "2416d0e0-25b6-485a-8612-723c279a3e21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:32.717136 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.717118 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2416d0e0-25b6-485a-8612-723c279a3e21-kube-api-access-xwxxb" pod "2416d0e0-25b6-485a-8612-723c279a3e21" (UID: "2416d0e0-25b6-485a-8612-723c279a3e21"). InnerVolumeSpecName "kube-api-access-xwxxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:32.717507 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.717490 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-metrics-certs" pod "2416d0e0-25b6-485a-8612-723c279a3e21" (UID: "2416d0e0-25b6-485a-8612-723c279a3e21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:32.816665 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.816634 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:32.816665 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.816665 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwxxb\" (UniqueName: \"kubernetes.io/projected/2416d0e0-25b6-485a-8612-723c279a3e21-kube-api-access-xwxxb\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:32.816891 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:32.816676 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2416d0e0-25b6-485a-8612-723c279a3e21-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:33.324326 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.324288 4477 generic.go:356] "Generic (PLEG): container finished" podID="2416d0e0-25b6-485a-8612-723c279a3e21" containerID="a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968" exitCode=0 Nov 01 21:13:33.324783 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.324346 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" Nov 01 21:13:33.324783 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.324337 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" event={"ID":"2416d0e0-25b6-485a-8612-723c279a3e21","Type":"ContainerDied","Data":"a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968"} Nov 01 21:13:33.324783 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.324484 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj" event={"ID":"2416d0e0-25b6-485a-8612-723c279a3e21","Type":"ContainerDied","Data":"dc48aa1fa25096a28dca1c7f2d033f2d1abe4a71f072a1e6047e31a7ea545ecb"} Nov 01 21:13:33.324783 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.324522 4477 scope.go:122] "RemoveContainer" containerID="a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968" Nov 01 21:13:33.334969 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.334949 4477 scope.go:122] "RemoveContainer" containerID="a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968" Nov 01 21:13:33.335316 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:13:33.335297 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968\": container with ID starting with a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968 not found: ID does not exist" containerID="a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968" Nov 01 21:13:33.335388 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.335326 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968"} err="failed to get container status \"a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968\": rpc error: code = NotFound desc = could not find container \"a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968\": container with ID starting with a108e2f9ae3bc84e8219c319a3e8ad380c54d6b8851555cc43a72dd16f02c968 not found: ID does not exist" Nov 01 21:13:33.488583 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.488548 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj"] Nov 01 21:13:33.537657 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:33.537618 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a6f5-6f86f77966-zlmgj"] Nov 01 21:13:34.507093 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:34.507056 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2416d0e0-25b6-485a-8612-723c279a3e21" path="/var/lib/kubelet/pods/2416d0e0-25b6-485a-8612-723c279a3e21/volumes" Nov 01 21:13:36.553851 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:13:36.553815 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:13:37.360006 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.359976 4477 generic.go:356] "Generic (PLEG): container finished" podID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerID="3efd509a119221d7c0f4d1a7d75da516102265711ed2e05109ce9acc94ab70a2" exitCode=0 Nov 01 21:13:37.360214 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.360186 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" event={"ID":"b3207b3d-5929-483f-b6a8-704e28cb09c2","Type":"ContainerDied","Data":"3efd509a119221d7c0f4d1a7d75da516102265711ed2e05109ce9acc94ab70a2"} Nov 01 21:13:37.464489 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.464464 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 21:13:37.572070 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572034 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-kube-api-access-vxpw7\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-kube-api-access-vxpw7\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572108 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-data\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572141 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-credential-socket\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572168 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-socket\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572229 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-token\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-token\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572275 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-envoy\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572299 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-podinfo\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572405 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3207b3d-5929-483f-b6a8-704e28cb09c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3207b3d-5929-483f-b6a8-704e28cb09c2-istiod-ca-cert\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572509 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-certs\") pod \"b3207b3d-5929-483f-b6a8-704e28cb09c2\" (UID: \"b3207b3d-5929-483f-b6a8-704e28cb09c2\") " Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572506 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-data" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572515 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-credential-socket" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.572558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572540 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-socket" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.573318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572849 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-certs" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.573318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.572856 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3207b3d-5929-483f-b6a8-704e28cb09c2-istiod-ca-cert" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:37.573318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.573037 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3207b3d-5929-483f-b6a8-704e28cb09c2-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.573318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.573055 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.573318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.573068 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.573318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.573115 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.573318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.573128 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.574811 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.574776 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-token" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:37.574811 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.574796 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-podinfo" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:13:37.575052 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.575034 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-kube-api-access-vxpw7" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "kube-api-access-vxpw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:37.575223 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.575204 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-envoy" pod "b3207b3d-5929-483f-b6a8-704e28cb09c2" (UID: "b3207b3d-5929-483f-b6a8-704e28cb09c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.674317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.674280 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxpw7\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-kube-api-access-vxpw7\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.674317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.674317 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.674547 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.674331 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.674547 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:37.674345 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3207b3d-5929-483f-b6a8-704e28cb09c2-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:38.367719 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:38.367688 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" Nov 01 21:13:38.367902 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:38.367683 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8" event={"ID":"b3207b3d-5929-483f-b6a8-704e28cb09c2","Type":"ContainerDied","Data":"feec700378279d2097adc5aad718d97e27257146fd219c9793e59adfc9a6d72f"} Nov 01 21:13:38.367902 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:38.367830 4477 scope.go:122] "RemoveContainer" containerID="3efd509a119221d7c0f4d1a7d75da516102265711ed2e05109ce9acc94ab70a2" Nov 01 21:13:38.582655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:38.582619 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8"] Nov 01 21:13:38.585620 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:38.585597 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-5bxs8"] Nov 01 21:13:40.503854 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:13:40.503770 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" path="/var/lib/kubelet/pods/b3207b3d-5929-483f-b6a8-704e28cb09c2/volumes" Nov 01 21:14:06.552883 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:06.552846 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:14:20.870445 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:20.870404 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:14:29.360136 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:29.360097 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz"] Nov 01 21:14:29.360608 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:29.360430 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" containerID="cri-o://8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1" gracePeriod=30 Nov 01 21:14:29.374575 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:29.374540 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp"] Nov 01 21:14:29.374762 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:29.374742 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" containerID="cri-o://542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c" gracePeriod=30 Nov 01 21:14:32.002223 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.002180 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c"] Nov 01 21:14:32.002740 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.002466 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" podUID="fdeeb41e-d051-4f52-a0d3-787e68a2d19a" containerName="maas-api" containerID="cri-o://23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd" gracePeriod=30 Nov 01 21:14:32.297266 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.297236 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:14:32.438645 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.438590 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls\") pod \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " Nov 01 21:14:32.438844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.438703 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-openshift-service-ca\") pod \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " Nov 01 21:14:32.438844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.438778 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-kube-api-access-dzf6t\" (UniqueName: \"kubernetes.io/projected/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-kube-api-access-dzf6t\") pod \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " Nov 01 21:14:32.438962 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.438939 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-metrics-certs\") pod \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\" (UID: \"fdeeb41e-d051-4f52-a0d3-787e68a2d19a\") " Nov 01 21:14:32.440700 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.440670 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-openshift-service-ca" pod "fdeeb41e-d051-4f52-a0d3-787e68a2d19a" (UID: "fdeeb41e-d051-4f52-a0d3-787e68a2d19a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:32.446247 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.446187 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-metrics-certs" pod "fdeeb41e-d051-4f52-a0d3-787e68a2d19a" (UID: "fdeeb41e-d051-4f52-a0d3-787e68a2d19a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:32.447469 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.447427 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls" pod "fdeeb41e-d051-4f52-a0d3-787e68a2d19a" (UID: "fdeeb41e-d051-4f52-a0d3-787e68a2d19a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:32.448173 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.448149 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-kube-api-access-dzf6t" pod "fdeeb41e-d051-4f52-a0d3-787e68a2d19a" (UID: "fdeeb41e-d051-4f52-a0d3-787e68a2d19a"). InnerVolumeSpecName "kube-api-access-dzf6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:32.541373 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.541241 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:32.541373 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.541274 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzf6t\" (UniqueName: \"kubernetes.io/projected/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-kube-api-access-dzf6t\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:32.541373 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.541290 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:32.541373 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.541300 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdeeb41e-d051-4f52-a0d3-787e68a2d19a-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:32.722986 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.722955 4477 generic.go:356] "Generic (PLEG): container finished" podID="fdeeb41e-d051-4f52-a0d3-787e68a2d19a" containerID="23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd" exitCode=0 Nov 01 21:14:32.723171 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.723007 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" Nov 01 21:14:32.723171 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.723039 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" event={"ID":"fdeeb41e-d051-4f52-a0d3-787e68a2d19a","Type":"ContainerDied","Data":"23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd"} Nov 01 21:14:32.723171 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.723117 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c" event={"ID":"fdeeb41e-d051-4f52-a0d3-787e68a2d19a","Type":"ContainerDied","Data":"9990415ce12f83c0a2d46ae0cc480dbb2de8e34759f1a8212e3f71688a7b454b"} Nov 01 21:14:32.723171 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.723142 4477 scope.go:122] "RemoveContainer" containerID="23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd" Nov 01 21:14:32.733247 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.733220 4477 scope.go:122] "RemoveContainer" containerID="23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd" Nov 01 21:14:32.733583 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:32.733563 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd\": container with ID starting with 23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd not found: ID does not exist" containerID="23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd" Nov 01 21:14:32.733642 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.733591 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd"} err="failed to get container status \"23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd\": rpc error: code = NotFound desc = could not find container \"23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd\": container with ID starting with 23d553fc161d4696fef102f61558e49b94803ab066c5005db928e561bbe5c6bd not found: ID does not exist" Nov 01 21:14:32.886585 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.886554 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c"] Nov 01 21:14:32.889527 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:32.889506 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-deff5e-7cf79d969b-vlx4c"] Nov 01 21:14:34.506833 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.506797 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdeeb41e-d051-4f52-a0d3-787e68a2d19a" path="/var/lib/kubelet/pods/fdeeb41e-d051-4f52-a0d3-787e68a2d19a/volumes" Nov 01 21:14:34.676169 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.676139 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:14:34.688572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.688550 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:14:34.741918 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.741880 4477 generic.go:356] "Generic (PLEG): container finished" podID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerID="8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1" exitCode=0 Nov 01 21:14:34.742069 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.741926 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" Nov 01 21:14:34.742069 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.741960 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" event={"ID":"fb621343-83ef-4b87-8c8e-e26e088560e5","Type":"ContainerDied","Data":"8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1"} Nov 01 21:14:34.742069 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.742016 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz" event={"ID":"fb621343-83ef-4b87-8c8e-e26e088560e5","Type":"ContainerDied","Data":"744e800c34358eae7d7fa9909cb1626b47d67e89761953bb4c9f58e9bbe2fb0a"} Nov 01 21:14:34.742069 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.742049 4477 scope.go:122] "RemoveContainer" containerID="8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1" Nov 01 21:14:34.743620 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.743596 4477 generic.go:356] "Generic (PLEG): container finished" podID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerID="542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c" exitCode=0 Nov 01 21:14:34.743893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.743641 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" Nov 01 21:14:34.743893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.743671 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" event={"ID":"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41","Type":"ContainerDied","Data":"542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c"} Nov 01 21:14:34.743893 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.743743 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp" event={"ID":"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41","Type":"ContainerDied","Data":"57461467916b43b8c8fda1bb0383b7f246081ec0f14353a7c42aa01bec56d61c"} Nov 01 21:14:34.752882 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.752866 4477 scope.go:122] "RemoveContainer" containerID="2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7" Nov 01 21:14:34.763013 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.762998 4477 scope.go:122] "RemoveContainer" containerID="8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1" Nov 01 21:14:34.763396 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:34.763371 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1\": container with ID starting with 8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1 not found: ID does not exist" containerID="8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1" Nov 01 21:14:34.763463 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.763410 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1"} err="failed to get container status \"8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1\": rpc error: code = NotFound desc = could not find container \"8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1\": container with ID starting with 8febf9ff578c40c30205b88e399b69ddcea3d5a20c649ab814799ec8557534d1 not found: ID does not exist" Nov 01 21:14:34.763463 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.763433 4477 scope.go:122] "RemoveContainer" containerID="2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7" Nov 01 21:14:34.763688 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:34.763670 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7\": container with ID starting with 2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7 not found: ID does not exist" containerID="2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7" Nov 01 21:14:34.763760 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.763700 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7"} err="failed to get container status \"2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7\": rpc error: code = NotFound desc = could not find container \"2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7\": container with ID starting with 2baaf27d7e3a626873030e7fcc2f5aac0e1d6e5d3e9174bfc202d87baead5ec7 not found: ID does not exist" Nov 01 21:14:34.763760 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.763721 4477 scope.go:122] "RemoveContainer" containerID="542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c" Nov 01 21:14:34.769021 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769000 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-certs\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.769121 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769057 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-socket\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769164 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769122 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-data\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769164 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769155 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-credential-socket\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769250 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769184 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-data\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.769301 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769262 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-envoy\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.769301 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769273 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-certs" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.769409 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769286 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-socket" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.769409 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769324 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-token\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-token\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769409 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769372 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-kube-api-access-xbk5l\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-kube-api-access-xbk5l\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.769409 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769386 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-data" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.769624 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769414 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-envoy\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769624 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769410 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-credential-socket" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.769624 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769446 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb621343-83ef-4b87-8c8e-e26e088560e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb621343-83ef-4b87-8c8e-e26e088560e5-istiod-ca-cert\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769624 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769490 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-podinfo\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769624 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769496 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-data" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.769624 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769547 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-token\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.769624 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769600 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-certs\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769977 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769647 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-socket\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.769977 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769690 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istiod-ca-cert\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.769977 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769733 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-podinfo\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.769977 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769769 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-kube-api-access-mckbq\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-kube-api-access-mckbq\") pod \"fb621343-83ef-4b87-8c8e-e26e088560e5\" (UID: \"fb621343-83ef-4b87-8c8e-e26e088560e5\") " Nov 01 21:14:34.769977 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769792 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb621343-83ef-4b87-8c8e-e26e088560e5-istiod-ca-cert" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:34.769977 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.769815 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-credential-socket\") pod \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\" (UID: \"3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41\") " Nov 01 21:14:34.770303 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770201 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-socket" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.770634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770369 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-credential-socket" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.770634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770377 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.770634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770413 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.770634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770430 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.770634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770447 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.770634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770470 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb621343-83ef-4b87-8c8e-e26e088560e5-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.770634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770483 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.770634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.770497 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.771335 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.771038 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-certs" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.771335 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.771212 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istiod-ca-cert" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:34.772235 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.772196 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-podinfo" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:14:34.773234 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.773202 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-podinfo" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:14:34.773405 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.773390 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-token" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:34.773523 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.773504 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-envoy" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.773582 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.773524 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-kube-api-access-xbk5l" pod "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" (UID: "3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41"). InnerVolumeSpecName "kube-api-access-xbk5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:34.773644 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.773628 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-kube-api-access-mckbq" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "kube-api-access-mckbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:34.773680 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.773635 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-token" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:34.774448 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.774433 4477 scope.go:122] "RemoveContainer" containerID="542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c" Nov 01 21:14:34.774732 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:34.774716 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c\": container with ID starting with 542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c not found: ID does not exist" containerID="542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c" Nov 01 21:14:34.774803 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.774741 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c"} err="failed to get container status \"542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c\": rpc error: code = NotFound desc = could not find container \"542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c\": container with ID starting with 542e7e0253b3d17a89c052679e242f38170a00d659aa148aaa296c8cebbda46c not found: ID does not exist" Nov 01 21:14:34.774985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.774971 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-envoy" pod "fb621343-83ef-4b87-8c8e-e26e088560e5" (UID: "fb621343-83ef-4b87-8c8e-e26e088560e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.871604 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871570 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871604 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871607 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871622 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871663 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mckbq\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-kube-api-access-mckbq\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871679 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871693 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871707 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871720 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbk5l\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-kube-api-access-xbk5l\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871733 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871747 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb621343-83ef-4b87-8c8e-e26e088560e5-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.871786 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:34.871760 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:35.169502 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:35.169459 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz"] Nov 01 21:14:35.172902 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:35.172876 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-h5bjz"] Nov 01 21:14:35.281451 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:35.281408 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp"] Nov 01 21:14:35.284600 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:35.284573 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-vpbgp"] Nov 01 21:14:36.504034 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:36.504001 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" path="/var/lib/kubelet/pods/3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41/volumes" Nov 01 21:14:36.504468 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:36.504450 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" path="/var/lib/kubelet/pods/fb621343-83ef-4b87-8c8e-e26e088560e5/volumes" Nov 01 21:14:36.553285 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:36.553250 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:14:40.946228 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:40.946194 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf"] Nov 01 21:14:40.946840 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:40.946395 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" containerID="cri-o://c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e" gracePeriod=30 Nov 01 21:14:46.228564 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.228498 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:14:46.314966 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.314934 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-podinfo\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315207 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.314988 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-credential-socket\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315207 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315013 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-kube-api-access-cjpfj\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-kube-api-access-cjpfj\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315207 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315034 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-token\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-token\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315207 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315171 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-socket\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315207 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315205 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-envoy\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315484 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315296 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-credential-socket" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:46.315484 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315307 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-certs\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315484 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315380 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-data\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315484 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315462 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dea6272-9dbc-435f-a85d-721b92bc7302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dea6272-9dbc-435f-a85d-721b92bc7302-istiod-ca-cert\") pod \"4dea6272-9dbc-435f-a85d-721b92bc7302\" (UID: \"4dea6272-9dbc-435f-a85d-721b92bc7302\") " Nov 01 21:14:46.315704 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315685 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-socket" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:46.315704 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315693 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-certs" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:46.315897 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315825 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-data" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:46.316064 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.315945 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dea6272-9dbc-435f-a85d-721b92bc7302-istiod-ca-cert" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:46.316200 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.316047 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.316354 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.316206 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.316354 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.316223 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.316354 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.316236 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.317716 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.317692 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-token" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:46.317821 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.317736 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-podinfo" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:14:46.318243 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.318223 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-envoy" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:46.318377 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.318357 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-kube-api-access-cjpfj" pod "4dea6272-9dbc-435f-a85d-721b92bc7302" (UID: "4dea6272-9dbc-435f-a85d-721b92bc7302"). InnerVolumeSpecName "kube-api-access-cjpfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:46.417899 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.417857 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.417899 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.417901 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dea6272-9dbc-435f-a85d-721b92bc7302-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.417899 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.417916 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.418242 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.417931 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjpfj\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-kube-api-access-cjpfj\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.418242 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.417947 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dea6272-9dbc-435f-a85d-721b92bc7302-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:46.833630 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.833599 4477 generic.go:356] "Generic (PLEG): container finished" podID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerID="c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e" exitCode=0 Nov 01 21:14:46.833815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.833663 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" event={"ID":"4dea6272-9dbc-435f-a85d-721b92bc7302","Type":"ContainerDied","Data":"c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e"} Nov 01 21:14:46.833815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.833703 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" event={"ID":"4dea6272-9dbc-435f-a85d-721b92bc7302","Type":"ContainerDied","Data":"91a2ba4a987b2870e81d629b85e2182445e1db3db04810929c09940e4dfca459"} Nov 01 21:14:46.833815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.833723 4477 scope.go:122] "RemoveContainer" containerID="c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e" Nov 01 21:14:46.833815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.833664 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf" Nov 01 21:14:46.845040 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.845018 4477 scope.go:122] "RemoveContainer" containerID="f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69" Nov 01 21:14:46.854667 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.854638 4477 scope.go:122] "RemoveContainer" containerID="c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e" Nov 01 21:14:46.854960 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:46.854941 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e\": container with ID starting with c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e not found: ID does not exist" containerID="c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e" Nov 01 21:14:46.855032 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.854971 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e"} err="failed to get container status \"c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e\": rpc error: code = NotFound desc = could not find container \"c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e\": container with ID starting with c13c12ebd4452172529f9dfe5b4f3f19f8de76993f384181ea5bab11643cc05e not found: ID does not exist" Nov 01 21:14:46.855032 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.854989 4477 scope.go:122] "RemoveContainer" containerID="f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69" Nov 01 21:14:46.855305 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:46.855283 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69\": container with ID starting with f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69 not found: ID does not exist" containerID="f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69" Nov 01 21:14:46.855359 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.855318 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69"} err="failed to get container status \"f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69\": rpc error: code = NotFound desc = could not find container \"f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69\": container with ID starting with f64b5b51f07d4c726c695939069260c44f3860d90d4c551da2836cd28fd44a69 not found: ID does not exist" Nov 01 21:14:46.997839 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:46.997805 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf"] Nov 01 21:14:47.048257 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:47.048222 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-deff5e-openshift-default-589bfb887f-hnvnf"] Nov 01 21:14:48.513709 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:48.513667 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" path="/var/lib/kubelet/pods/4dea6272-9dbc-435f-a85d-721b92bc7302/volumes" Nov 01 21:14:52.430747 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:52.430701 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh"] Nov 01 21:14:52.431293 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:52.430933 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" containerID="cri-o://67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4" gracePeriod=30 Nov 01 21:14:54.613894 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:54.613857 4477 patch_prober.go:29] interesting pod/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:14:54.614337 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:54.613926 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:14:57.677771 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.677740 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:14:57.846647 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.846614 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-certs\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.846852 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.846671 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-socket\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.846852 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.846709 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-token\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-token\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.846852 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.846759 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-envoy\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.846852 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.846790 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-data\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.846852 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.846836 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istiod-ca-cert\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.847157 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.846872 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-certs" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:57.847157 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.846879 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-credential-socket\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.847157 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847018 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-socket" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:57.847317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847269 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-data" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:57.847317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847304 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-credential-socket" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:57.847427 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847338 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istiod-ca-cert" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:57.847427 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847419 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-kube-api-access-clz2h\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-kube-api-access-clz2h\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.847533 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847480 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-podinfo\") pod \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\" (UID: \"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb\") " Nov 01 21:14:57.847995 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847904 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.847995 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847930 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.847995 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847943 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.847995 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847959 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.847995 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.847973 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.849569 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.849539 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-token" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:57.849868 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.849843 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-kube-api-access-clz2h" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "kube-api-access-clz2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:57.850319 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.850295 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-podinfo" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:14:57.851736 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.851715 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-envoy" pod "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" (UID: "cceb2a08-115d-42b7-ad54-5bf4bbdbbabb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:57.904442 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.904401 4477 generic.go:356] "Generic (PLEG): container finished" podID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerID="67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4" exitCode=0 Nov 01 21:14:57.904655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.904455 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" Nov 01 21:14:57.904655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.904495 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" event={"ID":"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb","Type":"ContainerDied","Data":"67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4"} Nov 01 21:14:57.904655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.904551 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh" event={"ID":"cceb2a08-115d-42b7-ad54-5bf4bbdbbabb","Type":"ContainerDied","Data":"3c9dc3f4d91cce55184978b21121f1de3f5dce3ee5fb4f12930743d031cba164"} Nov 01 21:14:57.904655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.904575 4477 scope.go:122] "RemoveContainer" containerID="67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4" Nov 01 21:14:57.915317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.915255 4477 scope.go:122] "RemoveContainer" containerID="d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b" Nov 01 21:14:57.924706 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.924683 4477 scope.go:122] "RemoveContainer" containerID="67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4" Nov 01 21:14:57.924993 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:57.924973 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4\": container with ID starting with 67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4 not found: ID does not exist" containerID="67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4" Nov 01 21:14:57.925089 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.925002 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4"} err="failed to get container status \"67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4\": rpc error: code = NotFound desc = could not find container \"67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4\": container with ID starting with 67c04097439de22d59d86624add3e301174c90bb475186225952e09958e8fcd4 not found: ID does not exist" Nov 01 21:14:57.925089 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.925020 4477 scope.go:122] "RemoveContainer" containerID="d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b" Nov 01 21:14:57.925280 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:14:57.925265 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b\": container with ID starting with d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b not found: ID does not exist" containerID="d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b" Nov 01 21:14:57.925347 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.925294 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b"} err="failed to get container status \"d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b\": rpc error: code = NotFound desc = could not find container \"d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b\": container with ID starting with d3026e24eedb8eec18c5aac811210b7ef6d707d933b3565fea424742c8f0d82b not found: ID does not exist" Nov 01 21:14:57.948604 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.948573 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.948604 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.948603 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.948815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.948614 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clz2h\" (UniqueName: \"kubernetes.io/projected/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-kube-api-access-clz2h\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.948815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:57.948623 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:58.068560 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:58.068526 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh"] Nov 01 21:14:58.074409 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:58.074376 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-044440-openshift-default-7f6ddf4485-wjzfh"] Nov 01 21:14:58.503954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:14:58.503922 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" path="/var/lib/kubelet/pods/cceb2a08-115d-42b7-ad54-5bf4bbdbbabb/volumes" Nov 01 21:15:00.192063 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.192024 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn"] Nov 01 21:15:00.193650 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193619 4477 cpu_manager.go:414] "containerMap: removing container" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" Nov 01 21:15:00.193650 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193649 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193690 4477 cpu_manager.go:414] "containerMap: removing container" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193702 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193718 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193726 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193742 4477 cpu_manager.go:414] "containerMap: removing container" podUID="2416d0e0-25b6-485a-8612-723c279a3e21" containerName="maas-api" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193749 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2416d0e0-25b6-485a-8612-723c279a3e21" containerName="maas-api" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193771 4477 cpu_manager.go:414] "containerMap: removing container" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193779 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193789 4477 cpu_manager.go:414] "containerMap: removing container" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193796 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193813 4477 cpu_manager.go:414] "containerMap: removing container" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193820 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193836 4477 cpu_manager.go:414] "containerMap: removing container" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193842 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193861 4477 cpu_manager.go:414] "containerMap: removing container" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193867 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" Nov 01 21:15:00.193874 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193880 4477 cpu_manager.go:414] "containerMap: removing container" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193887 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193901 4477 cpu_manager.go:414] "containerMap: removing container" podUID="06488271-ae4a-4f4e-8d6f-955a029ca19a" containerName="maas-api" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193907 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06488271-ae4a-4f4e-8d6f-955a029ca19a" containerName="maas-api" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193927 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193934 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193953 4477 cpu_manager.go:414] "containerMap: removing container" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193960 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193968 4477 cpu_manager.go:414] "containerMap: removing container" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.193974 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194000 4477 cpu_manager.go:414] "containerMap: removing container" podUID="fdeeb41e-d051-4f52-a0d3-787e68a2d19a" containerName="maas-api" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194007 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdeeb41e-d051-4f52-a0d3-787e68a2d19a" containerName="maas-api" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194023 4477 cpu_manager.go:414] "containerMap: removing container" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194029 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194268 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3207b3d-5929-483f-b6a8-704e28cb09c2" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194282 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194301 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdeeb41e-d051-4f52-a0d3-787e68a2d19a" containerName="maas-api" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194311 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c3028b-28a8-4a1c-9ce6-99ac8ab5622e" containerName="main" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194327 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1e1ac1-727a-48b3-b0e3-f199ba1f5d41" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194339 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194354 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194363 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2416d0e0-25b6-485a-8612-723c279a3e21" containerName="maas-api" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194378 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="cceb2a08-115d-42b7-ad54-5bf4bbdbbabb" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194387 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194402 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb621343-83ef-4b87-8c8e-e26e088560e5" containerName="istio-proxy" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194411 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0288ca11-f321-4f10-b59f-8f1a4e7c293b" containerName="main" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194425 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09423e7-1763-473d-b671-729f2d634515" containerName="main" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194434 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="06488271-ae4a-4f4e-8d6f-955a029ca19a" containerName="maas-api" Nov 01 21:15:00.194729 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.194448 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c13059-d000-41e5-90af-2536921d5165" containerName="istio-proxy" Nov 01 21:15:00.201603 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.201567 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:00.272969 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.272939 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgh7n\" (UniqueName: \"kubernetes.io/projected/872a3054-33bc-41ab-9f16-903afc70de64-kube-api-access-pgh7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn\" (UID: \"872a3054-33bc-41ab-9f16-903afc70de64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:00.273194 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.273008 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872a3054-33bc-41ab-9f16-903afc70de64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn\" (UID: \"872a3054-33bc-41ab-9f16-903afc70de64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:00.318331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.318067 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn"] Nov 01 21:15:00.362744 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.362713 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:15:00.374270 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.374230 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872a3054-33bc-41ab-9f16-903afc70de64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn\" (UID: \"872a3054-33bc-41ab-9f16-903afc70de64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:00.374433 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.374367 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgh7n\" (UniqueName: \"kubernetes.io/projected/872a3054-33bc-41ab-9f16-903afc70de64-kube-api-access-pgh7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn\" (UID: \"872a3054-33bc-41ab-9f16-903afc70de64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:00.374847 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.374829 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872a3054-33bc-41ab-9f16-903afc70de64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn\" (UID: \"872a3054-33bc-41ab-9f16-903afc70de64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:00.433009 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.432978 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgh7n\" (UniqueName: \"kubernetes.io/projected/872a3054-33bc-41ab-9f16-903afc70de64-kube-api-access-pgh7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn\" (UID: \"872a3054-33bc-41ab-9f16-903afc70de64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:00.516810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.516733 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:00.898224 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.898196 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn"] Nov 01 21:15:00.899727 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:15:00.899696 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872a3054_33bc_41ab_9f16_903afc70de64.slice/crio-2a8df872c25a50ba84ceda17ed97511ffbd2171e4deb5c5e10e886bcd8ee5752 WatchSource:0}: Error finding container 2a8df872c25a50ba84ceda17ed97511ffbd2171e4deb5c5e10e886bcd8ee5752: Status 404 returned error can't find the container with id 2a8df872c25a50ba84ceda17ed97511ffbd2171e4deb5c5e10e886bcd8ee5752 Nov 01 21:15:00.926900 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:00.926861 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" event={"ID":"872a3054-33bc-41ab-9f16-903afc70de64","Type":"ContainerStarted","Data":"2a8df872c25a50ba84ceda17ed97511ffbd2171e4deb5c5e10e886bcd8ee5752"} Nov 01 21:15:01.935227 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:01.935176 4477 generic.go:356] "Generic (PLEG): container finished" podID="872a3054-33bc-41ab-9f16-903afc70de64" containerID="fe7320d4fe1a391149c1db996e7c144540e746b099a2c16452c0a2da3e542861" exitCode=60 Nov 01 21:15:01.935665 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:01.935253 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" event={"ID":"872a3054-33bc-41ab-9f16-903afc70de64","Type":"ContainerDied","Data":"fe7320d4fe1a391149c1db996e7c144540e746b099a2c16452c0a2da3e542861"} Nov 01 21:15:01.935665 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:01.935551 4477 scope.go:122] "RemoveContainer" containerID="fe7320d4fe1a391149c1db996e7c144540e746b099a2c16452c0a2da3e542861" Nov 01 21:15:02.944514 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:02.944478 4477 generic.go:356] "Generic (PLEG): container finished" podID="872a3054-33bc-41ab-9f16-903afc70de64" containerID="251084e9045ce3da3d35178a3dc1116a52cf0025bcf6c7141f4933d176209510" exitCode=60 Nov 01 21:15:02.945004 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:02.944526 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" event={"ID":"872a3054-33bc-41ab-9f16-903afc70de64","Type":"ContainerDied","Data":"251084e9045ce3da3d35178a3dc1116a52cf0025bcf6c7141f4933d176209510"} Nov 01 21:15:02.945004 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:02.944602 4477 scope.go:122] "RemoveContainer" containerID="fe7320d4fe1a391149c1db996e7c144540e746b099a2c16452c0a2da3e542861" Nov 01 21:15:02.945004 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:02.944873 4477 scope.go:122] "RemoveContainer" containerID="251084e9045ce3da3d35178a3dc1116a52cf0025bcf6c7141f4933d176209510" Nov 01 21:15:02.945215 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:02.945188 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn_odh-ai-gateway-infra(872a3054-33bc-41ab-9f16-903afc70de64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" podUID="872a3054-33bc-41ab-9f16-903afc70de64" Nov 01 21:15:03.622023 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:03.621984 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:15:03.964029 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:03.963953 4477 scope.go:122] "RemoveContainer" containerID="251084e9045ce3da3d35178a3dc1116a52cf0025bcf6c7141f4933d176209510" Nov 01 21:15:03.964423 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:03.964220 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn_odh-ai-gateway-infra(872a3054-33bc-41ab-9f16-903afc70de64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" podUID="872a3054-33bc-41ab-9f16-903afc70de64" Nov 01 21:15:06.560141 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:06.560097 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:15:13.497560 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:13.497530 4477 scope.go:122] "RemoveContainer" containerID="251084e9045ce3da3d35178a3dc1116a52cf0025bcf6c7141f4933d176209510" Nov 01 21:15:14.028003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:14.027967 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" event={"ID":"872a3054-33bc-41ab-9f16-903afc70de64","Type":"ContainerStarted","Data":"2140577af46effd4bfa4c49d1e0f7bd3863abc617c8c4871a5e31973ef517c1b"} Nov 01 21:15:14.142011 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:14.141949 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" podStartSLOduration=14.141930265 podStartE2EDuration="14.141930265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:14.139352561 +0000 UTC m=+2667.789404118" watchObservedRunningTime="2026-11-01 21:15:14.141930265 +0000 UTC m=+2667.791981812" Nov 01 21:15:15.036587 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:15.036550 4477 generic.go:356] "Generic (PLEG): container finished" podID="872a3054-33bc-41ab-9f16-903afc70de64" containerID="2140577af46effd4bfa4c49d1e0f7bd3863abc617c8c4871a5e31973ef517c1b" exitCode=60 Nov 01 21:15:15.037133 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:15.036625 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" event={"ID":"872a3054-33bc-41ab-9f16-903afc70de64","Type":"ContainerDied","Data":"2140577af46effd4bfa4c49d1e0f7bd3863abc617c8c4871a5e31973ef517c1b"} Nov 01 21:15:15.037133 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:15.036688 4477 scope.go:122] "RemoveContainer" containerID="251084e9045ce3da3d35178a3dc1116a52cf0025bcf6c7141f4933d176209510" Nov 01 21:15:15.037133 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:15.037128 4477 scope.go:122] "RemoveContainer" containerID="2140577af46effd4bfa4c49d1e0f7bd3863abc617c8c4871a5e31973ef517c1b" Nov 01 21:15:15.037477 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:15.037446 4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn_odh-ai-gateway-infra(872a3054-33bc-41ab-9f16-903afc70de64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" podUID="872a3054-33bc-41ab-9f16-903afc70de64" Nov 01 21:15:15.151968 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:15.151932 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn"] Nov 01 21:15:16.190597 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:16.190572 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:16.227569 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:16.227515 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872a3054-33bc-41ab-9f16-903afc70de64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872a3054-33bc-41ab-9f16-903afc70de64-maas-api-service-ca\") pod \"872a3054-33bc-41ab-9f16-903afc70de64\" (UID: \"872a3054-33bc-41ab-9f16-903afc70de64\") " Nov 01 21:15:16.227792 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:16.227695 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872a3054-33bc-41ab-9f16-903afc70de64-kube-api-access-pgh7n\" (UniqueName: \"kubernetes.io/projected/872a3054-33bc-41ab-9f16-903afc70de64-kube-api-access-pgh7n\") pod \"872a3054-33bc-41ab-9f16-903afc70de64\" (UID: \"872a3054-33bc-41ab-9f16-903afc70de64\") " Nov 01 21:15:16.227953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:16.227929 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872a3054-33bc-41ab-9f16-903afc70de64-maas-api-service-ca" pod "872a3054-33bc-41ab-9f16-903afc70de64" (UID: "872a3054-33bc-41ab-9f16-903afc70de64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:16.228168 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:16.228151 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872a3054-33bc-41ab-9f16-903afc70de64-maas-api-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:16.229945 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:16.229923 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872a3054-33bc-41ab-9f16-903afc70de64-kube-api-access-pgh7n" pod "872a3054-33bc-41ab-9f16-903afc70de64" (UID: "872a3054-33bc-41ab-9f16-903afc70de64"). InnerVolumeSpecName "kube-api-access-pgh7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:16.329400 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:16.329361 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgh7n\" (UniqueName: \"kubernetes.io/projected/872a3054-33bc-41ab-9f16-903afc70de64-kube-api-access-pgh7n\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:17.053229 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:17.053200 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" Nov 01 21:15:17.053229 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:17.053222 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn" event={"ID":"872a3054-33bc-41ab-9f16-903afc70de64","Type":"ContainerDied","Data":"2a8df872c25a50ba84ceda17ed97511ffbd2171e4deb5c5e10e886bcd8ee5752"} Nov 01 21:15:17.053509 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:17.053290 4477 scope.go:122] "RemoveContainer" containerID="2140577af46effd4bfa4c49d1e0f7bd3863abc617c8c4871a5e31973ef517c1b" Nov 01 21:15:17.218513 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:17.218475 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn"] Nov 01 21:15:17.221512 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:17.221483 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892795-4l8tn"] Nov 01 21:15:18.504570 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:18.504541 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872a3054-33bc-41ab-9f16-903afc70de64" path="/var/lib/kubelet/pods/872a3054-33bc-41ab-9f16-903afc70de64/volumes" Nov 01 21:15:18.885737 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:18.885701 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6"] Nov 01 21:15:18.885997 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:18.885951 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" containerID="cri-o://22dfeded88e18f8ed4238a4d1a184f985562ecc0974472b19ddbd5febadbcc83" gracePeriod=30 Nov 01 21:15:21.744365 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:21.744332 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r"] Nov 01 21:15:21.744888 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:21.744614 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" podUID="bb31a6c3-1a59-4b34-b906-d1a335333566" containerName="maas-api" containerID="cri-o://4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28" gracePeriod=30 Nov 01 21:15:22.032226 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.032190 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 21:15:22.087319 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.087281 4477 generic.go:356] "Generic (PLEG): container finished" podID="bb31a6c3-1a59-4b34-b906-d1a335333566" containerID="4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28" exitCode=0 Nov 01 21:15:22.087530 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.087484 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" event={"ID":"bb31a6c3-1a59-4b34-b906-d1a335333566","Type":"ContainerDied","Data":"4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28"} Nov 01 21:15:22.087603 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.087533 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" Nov 01 21:15:22.087603 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.087549 4477 scope.go:122] "RemoveContainer" containerID="4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28" Nov 01 21:15:22.087711 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.087536 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r" event={"ID":"bb31a6c3-1a59-4b34-b906-d1a335333566","Type":"ContainerDied","Data":"d7a96a48a1e78190864ed9662593f1015ed996e18400288129b6f4391ae381f6"} Nov 01 21:15:22.093042 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.093019 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls\") pod \"bb31a6c3-1a59-4b34-b906-d1a335333566\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " Nov 01 21:15:22.093192 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.093117 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb31a6c3-1a59-4b34-b906-d1a335333566-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb31a6c3-1a59-4b34-b906-d1a335333566-openshift-service-ca\") pod \"bb31a6c3-1a59-4b34-b906-d1a335333566\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " Nov 01 21:15:22.093192 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.093165 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb31a6c3-1a59-4b34-b906-d1a335333566-kube-api-access-wkxvs\" (UniqueName: \"kubernetes.io/projected/bb31a6c3-1a59-4b34-b906-d1a335333566-kube-api-access-wkxvs\") pod \"bb31a6c3-1a59-4b34-b906-d1a335333566\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " Nov 01 21:15:22.093312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.093227 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-metrics-certs\") pod \"bb31a6c3-1a59-4b34-b906-d1a335333566\" (UID: \"bb31a6c3-1a59-4b34-b906-d1a335333566\") " Nov 01 21:15:22.093662 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.093459 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb31a6c3-1a59-4b34-b906-d1a335333566-openshift-service-ca" pod "bb31a6c3-1a59-4b34-b906-d1a335333566" (UID: "bb31a6c3-1a59-4b34-b906-d1a335333566"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:22.093780 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.093720 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb31a6c3-1a59-4b34-b906-d1a335333566-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:22.095757 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.095729 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-metrics-certs" pod "bb31a6c3-1a59-4b34-b906-d1a335333566" (UID: "bb31a6c3-1a59-4b34-b906-d1a335333566"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:22.095861 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.095836 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls" pod "bb31a6c3-1a59-4b34-b906-d1a335333566" (UID: "bb31a6c3-1a59-4b34-b906-d1a335333566"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:22.099229 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.099212 4477 scope.go:122] "RemoveContainer" containerID="4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28" Nov 01 21:15:22.099506 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.099476 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb31a6c3-1a59-4b34-b906-d1a335333566-kube-api-access-wkxvs" pod "bb31a6c3-1a59-4b34-b906-d1a335333566" (UID: "bb31a6c3-1a59-4b34-b906-d1a335333566"). InnerVolumeSpecName "kube-api-access-wkxvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:22.099612 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:22.099529 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28\": container with ID starting with 4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28 not found: ID does not exist" containerID="4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28" Nov 01 21:15:22.099612 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.099553 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28"} err="failed to get container status \"4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28\": rpc error: code = NotFound desc = could not find container \"4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28\": container with ID starting with 4bf9221d3d03cd4dd447cc363548027081b4ca3271b3e198b7466798b655ce28 not found: ID does not exist" Nov 01 21:15:22.194538 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.194498 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkxvs\" (UniqueName: \"kubernetes.io/projected/bb31a6c3-1a59-4b34-b906-d1a335333566-kube-api-access-wkxvs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:22.194538 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.194532 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:22.194538 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.194543 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb31a6c3-1a59-4b34-b906-d1a335333566-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:22.507306 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.507270 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r"] Nov 01 21:15:22.510196 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:22.510170 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dce296-59797c589b-8z52r"] Nov 01 21:15:23.127256 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.127224 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz"] Nov 01 21:15:23.127669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.127469 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" podUID="fcb2250c-fc1b-4982-82d5-678fb31f35f3" containerName="maas-api" containerID="cri-o://7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0" gracePeriod=30 Nov 01 21:15:23.441919 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.441621 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:15:23.511060 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.511028 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls\") pod \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " Nov 01 21:15:23.511270 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.511188 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-metrics-certs\") pod \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " Nov 01 21:15:23.511270 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.511217 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb2250c-fc1b-4982-82d5-678fb31f35f3-kube-api-access-r2cb6\" (UniqueName: \"kubernetes.io/projected/fcb2250c-fc1b-4982-82d5-678fb31f35f3-kube-api-access-r2cb6\") pod \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " Nov 01 21:15:23.511412 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.511288 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb2250c-fc1b-4982-82d5-678fb31f35f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb2250c-fc1b-4982-82d5-678fb31f35f3-openshift-service-ca\") pod \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\" (UID: \"fcb2250c-fc1b-4982-82d5-678fb31f35f3\") " Nov 01 21:15:23.511915 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.511881 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb2250c-fc1b-4982-82d5-678fb31f35f3-openshift-service-ca" pod "fcb2250c-fc1b-4982-82d5-678fb31f35f3" (UID: "fcb2250c-fc1b-4982-82d5-678fb31f35f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:23.513438 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.513409 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls" pod "fcb2250c-fc1b-4982-82d5-678fb31f35f3" (UID: "fcb2250c-fc1b-4982-82d5-678fb31f35f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:23.513671 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.513643 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb2250c-fc1b-4982-82d5-678fb31f35f3-kube-api-access-r2cb6" pod "fcb2250c-fc1b-4982-82d5-678fb31f35f3" (UID: "fcb2250c-fc1b-4982-82d5-678fb31f35f3"). InnerVolumeSpecName "kube-api-access-r2cb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:23.513793 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.513714 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-metrics-certs" pod "fcb2250c-fc1b-4982-82d5-678fb31f35f3" (UID: "fcb2250c-fc1b-4982-82d5-678fb31f35f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:23.612264 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.612231 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:23.612264 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.612263 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2cb6\" (UniqueName: \"kubernetes.io/projected/fcb2250c-fc1b-4982-82d5-678fb31f35f3-kube-api-access-r2cb6\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:23.612489 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.612275 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb2250c-fc1b-4982-82d5-678fb31f35f3-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:23.612489 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:23.612284 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb2250c-fc1b-4982-82d5-678fb31f35f3-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.102448 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.102407 4477 generic.go:356] "Generic (PLEG): container finished" podID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerID="22dfeded88e18f8ed4238a4d1a184f985562ecc0974472b19ddbd5febadbcc83" exitCode=0 Nov 01 21:15:24.102448 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.102440 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" event={"ID":"fd279d92-f03e-4a2e-8a0d-b785efaeace6","Type":"ContainerDied","Data":"22dfeded88e18f8ed4238a4d1a184f985562ecc0974472b19ddbd5febadbcc83"} Nov 01 21:15:24.103877 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.103852 4477 generic.go:356] "Generic (PLEG): container finished" podID="fcb2250c-fc1b-4982-82d5-678fb31f35f3" containerID="7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0" exitCode=0 Nov 01 21:15:24.104035 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.103914 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" Nov 01 21:15:24.104035 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.103916 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" event={"ID":"fcb2250c-fc1b-4982-82d5-678fb31f35f3","Type":"ContainerDied","Data":"7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0"} Nov 01 21:15:24.104161 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.104038 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz" event={"ID":"fcb2250c-fc1b-4982-82d5-678fb31f35f3","Type":"ContainerDied","Data":"e8d0355a1aab53131399a7655b865f59a96b7a192edb0f3ccdb2afb5fe593bd7"} Nov 01 21:15:24.104161 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.104061 4477 scope.go:122] "RemoveContainer" containerID="7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0" Nov 01 21:15:24.117152 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.117069 4477 scope.go:122] "RemoveContainer" containerID="7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0" Nov 01 21:15:24.117918 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:24.117798 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0\": container with ID starting with 7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0 not found: ID does not exist" containerID="7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0" Nov 01 21:15:24.117918 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.117864 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0"} err="failed to get container status \"7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0\": rpc error: code = NotFound desc = could not find container \"7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0\": container with ID starting with 7fe75f66d918613d7586f1d6a127be467061082d1eaacac5de8e7f35729f15b0 not found: ID does not exist" Nov 01 21:15:24.195140 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.195116 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 21:15:24.268878 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.268843 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz"] Nov 01 21:15:24.318467 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318427 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-credential-socket\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.318669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318500 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-data\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.318669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318542 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-envoy\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.318669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318647 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-socket\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.318850 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318685 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-podinfo\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.318850 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318722 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-credential-socket" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:24.318850 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318754 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istiod-ca-cert\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.318850 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318788 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-token\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-token\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.318850 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318813 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-certs\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.318850 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318814 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-data" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:24.319200 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.318941 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-socket" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:24.319200 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319051 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-certs" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:24.319317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319196 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istiod-ca-cert" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:24.319317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319244 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-kube-api-access-rq54w\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-kube-api-access-rq54w\") pod \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\" (UID: \"fd279d92-f03e-4a2e-8a0d-b785efaeace6\") " Nov 01 21:15:24.319750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319645 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.319750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319674 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.319750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319690 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.319750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319705 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.319750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319718 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.320293 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.319823 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-670df9-7b5c876444-dlsnz"] Nov 01 21:15:24.322109 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.321605 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-podinfo" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:24.322109 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.321674 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-envoy" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:24.322109 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.321669 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-kube-api-access-rq54w" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "kube-api-access-rq54w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:24.322109 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.321702 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-token" pod "fd279d92-f03e-4a2e-8a0d-b785efaeace6" (UID: "fd279d92-f03e-4a2e-8a0d-b785efaeace6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:24.420506 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.420480 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.420506 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.420505 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.420676 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.420520 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.420676 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.420535 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq54w\" (UniqueName: \"kubernetes.io/projected/fd279d92-f03e-4a2e-8a0d-b785efaeace6-kube-api-access-rq54w\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:24.502831 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.502757 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb31a6c3-1a59-4b34-b906-d1a335333566" path="/var/lib/kubelet/pods/bb31a6c3-1a59-4b34-b906-d1a335333566/volumes" Nov 01 21:15:24.503243 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:24.503229 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb2250c-fc1b-4982-82d5-678fb31f35f3" path="/var/lib/kubelet/pods/fcb2250c-fc1b-4982-82d5-678fb31f35f3/volumes" Nov 01 21:15:25.111582 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:25.111543 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" Nov 01 21:15:25.111779 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:25.111542 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6" event={"ID":"fd279d92-f03e-4a2e-8a0d-b785efaeace6","Type":"ContainerDied","Data":"294aaee8c45a078da79037b5312807c92950f33cdd4974e83b7b2d2c4557ec19"} Nov 01 21:15:25.111779 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:25.111694 4477 scope.go:122] "RemoveContainer" containerID="22dfeded88e18f8ed4238a4d1a184f985562ecc0974472b19ddbd5febadbcc83" Nov 01 21:15:25.275737 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:25.275703 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6"] Nov 01 21:15:25.325882 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:25.325842 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-sh9p6"] Nov 01 21:15:26.503888 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:26.503853 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" path="/var/lib/kubelet/pods/fd279d92-f03e-4a2e-8a0d-b785efaeace6/volumes" Nov 01 21:15:32.187655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:32.187618 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq"] Nov 01 21:15:32.188120 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:32.187843 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" containerID="cri-o://26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f" gracePeriod=30 Nov 01 21:15:32.237349 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:32.237311 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw"] Nov 01 21:15:32.237550 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:32.237529 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" containerID="cri-o://f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a" gracePeriod=30 Nov 01 21:15:34.896411 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:34.896375 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs"] Nov 01 21:15:34.896826 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:34.896601 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" podUID="8578e043-d4a0-4968-a7f7-b9fa052c8fb1" containerName="maas-api" containerID="cri-o://ade76562919d18e6d08213dbd11cb8e8e9c10f4c226549c537225944e6bb87b3" gracePeriod=30 Nov 01 21:15:35.181957 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.181927 4477 generic.go:356] "Generic (PLEG): container finished" podID="8578e043-d4a0-4968-a7f7-b9fa052c8fb1" containerID="ade76562919d18e6d08213dbd11cb8e8e9c10f4c226549c537225944e6bb87b3" exitCode=0 Nov 01 21:15:35.182134 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.182019 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" event={"ID":"8578e043-d4a0-4968-a7f7-b9fa052c8fb1","Type":"ContainerDied","Data":"ade76562919d18e6d08213dbd11cb8e8e9c10f4c226549c537225944e6bb87b3"} Nov 01 21:15:35.203388 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.203362 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:15:35.339951 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.339916 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-openshift-service-ca\") pod \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " Nov 01 21:15:35.340168 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.340044 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-metrics-certs\") pod \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " Nov 01 21:15:35.340168 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.340133 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-kube-api-access-mnzwp\" (UniqueName: \"kubernetes.io/projected/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-kube-api-access-mnzwp\") pod \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " Nov 01 21:15:35.340303 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.340200 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls\") pod \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\" (UID: \"8578e043-d4a0-4968-a7f7-b9fa052c8fb1\") " Nov 01 21:15:35.340557 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.340525 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-openshift-service-ca" pod "8578e043-d4a0-4968-a7f7-b9fa052c8fb1" (UID: "8578e043-d4a0-4968-a7f7-b9fa052c8fb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:35.342508 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.342478 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-metrics-certs" pod "8578e043-d4a0-4968-a7f7-b9fa052c8fb1" (UID: "8578e043-d4a0-4968-a7f7-b9fa052c8fb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:35.342842 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.342810 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls" pod "8578e043-d4a0-4968-a7f7-b9fa052c8fb1" (UID: "8578e043-d4a0-4968-a7f7-b9fa052c8fb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:35.342942 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.342873 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-kube-api-access-mnzwp" pod "8578e043-d4a0-4968-a7f7-b9fa052c8fb1" (UID: "8578e043-d4a0-4968-a7f7-b9fa052c8fb1"). InnerVolumeSpecName "kube-api-access-mnzwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:35.441670 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.441591 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:35.441670 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.441623 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzwp\" (UniqueName: \"kubernetes.io/projected/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-kube-api-access-mnzwp\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:35.441670 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.441634 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:35.441670 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:35.441642 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8578e043-d4a0-4968-a7f7-b9fa052c8fb1-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:36.189989 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:36.189957 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" Nov 01 21:15:36.190523 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:36.189996 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs" event={"ID":"8578e043-d4a0-4968-a7f7-b9fa052c8fb1","Type":"ContainerDied","Data":"b8fa77ee391789a5a2c65d0ccd49c426af21a3e1f18e70fe2e02709f7cb51256"} Nov 01 21:15:36.190523 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:36.190057 4477 scope.go:122] "RemoveContainer" containerID="ade76562919d18e6d08213dbd11cb8e8e9c10f4c226549c537225944e6bb87b3" Nov 01 21:15:36.354183 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:36.354148 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs"] Nov 01 21:15:36.403500 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:36.403464 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a41242-5cd46b874b-kv6fs"] Nov 01 21:15:36.503559 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:36.503484 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8578e043-d4a0-4968-a7f7-b9fa052c8fb1" path="/var/lib/kubelet/pods/8578e043-d4a0-4968-a7f7-b9fa052c8fb1/volumes" Nov 01 21:15:36.552037 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:36.552001 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:15:37.497661 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.497633 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:15:37.539859 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.539829 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:15:37.604543 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.604514 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc"] Nov 01 21:15:37.667540 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.667505 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-envoy\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.668279 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.667936 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-certs\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.668279 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668214 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-certs" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.668428 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668282 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-token\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-token\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.668428 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668353 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-podinfo\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.668428 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668385 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-envoy\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.668428 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668421 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-istio-token\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-istio-token\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.668660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668451 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-kube-api-access-dtjxv\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-kube-api-access-dtjxv\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.668660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668482 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-credential-socket\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.668660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668514 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-credential-socket\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.668660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668543 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-data\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.668660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668587 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istiod-ca-cert\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.668660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668630 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-socket\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.668660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668660 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-socket\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.669050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668698 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-data\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.669050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668730 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f591f6-2c37-47df-b609-08d801eb30fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f591f6-2c37-47df-b609-08d801eb30fa-istiod-ca-cert\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.669050 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.668989 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f591f6-2c37-47df-b609-08d801eb30fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f591f6-2c37-47df-b609-08d801eb30fa-istio-podinfo\") pod \"22f591f6-2c37-47df-b609-08d801eb30fa\" (UID: \"22f591f6-2c37-47df-b609-08d801eb30fa\") " Nov 01 21:15:37.669325 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.669051 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-certs\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.669325 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.669166 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-kube-api-access-zcrrh\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-kube-api-access-zcrrh\") pod \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\" (UID: \"19f054e8-f0c0-4968-b8b2-c76fafb1876d\") " Nov 01 21:15:37.670264 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.670236 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-credential-socket" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.670395 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.670244 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-data" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.670395 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.670276 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istiod-ca-cert" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:37.670395 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.670381 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-socket" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.670572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.670528 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-socket" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.670572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.670541 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-credential-socket" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.670962 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.670930 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-certs" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.671116 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.671095 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-data" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.671512 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.671487 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f591f6-2c37-47df-b609-08d801eb30fa-istiod-ca-cert" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:37.671603 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.671523 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-envoy" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.674125 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.674025 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-podinfo" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:37.674125 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.674056 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-istio-token" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:37.674305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.674129 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-kube-api-access-zcrrh" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "kube-api-access-zcrrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:37.675092 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.675057 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-kube-api-access-dtjxv" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "kube-api-access-dtjxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:37.675808 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.675774 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f591f6-2c37-47df-b609-08d801eb30fa-istio-podinfo" pod "22f591f6-2c37-47df-b609-08d801eb30fa" (UID: "22f591f6-2c37-47df-b609-08d801eb30fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:37.677496 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.677473 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-envoy" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.678684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.678658 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-token" pod "19f054e8-f0c0-4968-b8b2-c76fafb1876d" (UID: "19f054e8-f0c0-4968-b8b2-c76fafb1876d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:37.770611 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770529 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f591f6-2c37-47df-b609-08d801eb30fa-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770611 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770558 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770611 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770568 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcrrh\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-kube-api-access-zcrrh\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770611 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770581 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770611 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770596 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770611 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770606 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770611 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770614 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770623 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770633 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770641 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtjxv\" (UniqueName: \"kubernetes.io/projected/22f591f6-2c37-47df-b609-08d801eb30fa-kube-api-access-dtjxv\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770649 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770661 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770673 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770680 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770688 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770695 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f591f6-2c37-47df-b609-08d801eb30fa-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770702 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f054e8-f0c0-4968-b8b2-c76fafb1876d-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.770953 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:37.770710 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f591f6-2c37-47df-b609-08d801eb30fa-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:38.206021 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.205987 4477 generic.go:356] "Generic (PLEG): container finished" podID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerID="26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f" exitCode=0 Nov 01 21:15:38.206201 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.206053 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" Nov 01 21:15:38.206201 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.206069 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" event={"ID":"19f054e8-f0c0-4968-b8b2-c76fafb1876d","Type":"ContainerDied","Data":"26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f"} Nov 01 21:15:38.206201 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.206168 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq" event={"ID":"19f054e8-f0c0-4968-b8b2-c76fafb1876d","Type":"ContainerDied","Data":"485a0be6eef08a695016bad5795ae0580f171589ab54097504d917ded9528304"} Nov 01 21:15:38.206201 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.206194 4477 scope.go:122] "RemoveContainer" containerID="26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f" Nov 01 21:15:38.207837 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.207809 4477 generic.go:356] "Generic (PLEG): container finished" podID="22f591f6-2c37-47df-b609-08d801eb30fa" containerID="f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a" exitCode=0 Nov 01 21:15:38.207989 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.207836 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" event={"ID":"22f591f6-2c37-47df-b609-08d801eb30fa","Type":"ContainerDied","Data":"f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a"} Nov 01 21:15:38.207989 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.207874 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" Nov 01 21:15:38.207989 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.207896 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw" event={"ID":"22f591f6-2c37-47df-b609-08d801eb30fa","Type":"ContainerDied","Data":"1b019613c731655ae509f06aa203f0130aa194967de89af73040183729aa53ce"} Nov 01 21:15:38.224286 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.224263 4477 scope.go:122] "RemoveContainer" containerID="26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f" Nov 01 21:15:38.224627 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:38.224606 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f\": container with ID starting with 26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f not found: ID does not exist" containerID="26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f" Nov 01 21:15:38.224684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.224639 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f"} err="failed to get container status \"26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f\": rpc error: code = NotFound desc = could not find container \"26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f\": container with ID starting with 26dea50c489415b71e2a1874d75e9d2a9722399220edd0b3ebe329d5692ba37f not found: ID does not exist" Nov 01 21:15:38.224684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.224657 4477 scope.go:122] "RemoveContainer" containerID="f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a" Nov 01 21:15:38.235220 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.235200 4477 scope.go:122] "RemoveContainer" containerID="f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a" Nov 01 21:15:38.235563 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:38.235545 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a\": container with ID starting with f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a not found: ID does not exist" containerID="f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a" Nov 01 21:15:38.235623 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.235573 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a"} err="failed to get container status \"f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a\": rpc error: code = NotFound desc = could not find container \"f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a\": container with ID starting with f740661de7ec358d23e725a2c251ad9e0ea3728a860f21efb66b78a816aa353a not found: ID does not exist" Nov 01 21:15:38.372964 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.372926 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw"] Nov 01 21:15:38.377235 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.377203 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-pg8rw"] Nov 01 21:15:38.484216 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.484133 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq"] Nov 01 21:15:38.490025 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.489994 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m6rnq"] Nov 01 21:15:38.502715 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.502691 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" path="/var/lib/kubelet/pods/19f054e8-f0c0-4968-b8b2-c76fafb1876d/volumes" Nov 01 21:15:38.503132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:38.503119 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" path="/var/lib/kubelet/pods/22f591f6-2c37-47df-b609-08d801eb30fa/volumes" Nov 01 21:15:45.091098 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:45.091046 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:15:52.624843 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.624781 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" containerID="cri-o://9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4" gracePeriod=45 Nov 01 21:15:52.880147 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.880086 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 21:15:52.914217 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914177 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-dshm\") pod \"00b8771f-982e-4022-a1de-ab6373b7dab1\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " Nov 01 21:15:52.914217 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914230 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-model-cache\") pod \"00b8771f-982e-4022-a1de-ab6373b7dab1\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " Nov 01 21:15:52.914500 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914366 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00b8771f-982e-4022-a1de-ab6373b7dab1-tls-certs\" (UniqueName: \"kubernetes.io/secret/00b8771f-982e-4022-a1de-ab6373b7dab1-tls-certs\") pod \"00b8771f-982e-4022-a1de-ab6373b7dab1\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " Nov 01 21:15:52.914500 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914398 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-tmp-dir\") pod \"00b8771f-982e-4022-a1de-ab6373b7dab1\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " Nov 01 21:15:52.914500 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914458 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b8771f-982e-4022-a1de-ab6373b7dab1-kube-api-access-82rbm\" (UniqueName: \"kubernetes.io/projected/00b8771f-982e-4022-a1de-ab6373b7dab1-kube-api-access-82rbm\") pod \"00b8771f-982e-4022-a1de-ab6373b7dab1\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " Nov 01 21:15:52.914683 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914553 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-home\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-home\") pod \"00b8771f-982e-4022-a1de-ab6373b7dab1\" (UID: \"00b8771f-982e-4022-a1de-ab6373b7dab1\") " Nov 01 21:15:52.914683 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914575 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-model-cache" pod "00b8771f-982e-4022-a1de-ab6373b7dab1" (UID: "00b8771f-982e-4022-a1de-ab6373b7dab1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:52.914828 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914792 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-tmp-dir" pod "00b8771f-982e-4022-a1de-ab6373b7dab1" (UID: "00b8771f-982e-4022-a1de-ab6373b7dab1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:52.914978 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914906 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:52.914978 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.914925 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:52.915169 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.915100 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-home" pod "00b8771f-982e-4022-a1de-ab6373b7dab1" (UID: "00b8771f-982e-4022-a1de-ab6373b7dab1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:52.917484 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.917452 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-dshm" pod "00b8771f-982e-4022-a1de-ab6373b7dab1" (UID: "00b8771f-982e-4022-a1de-ab6373b7dab1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:52.917622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.917541 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b8771f-982e-4022-a1de-ab6373b7dab1-kube-api-access-82rbm" pod "00b8771f-982e-4022-a1de-ab6373b7dab1" (UID: "00b8771f-982e-4022-a1de-ab6373b7dab1"). InnerVolumeSpecName "kube-api-access-82rbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:52.917738 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:52.917720 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b8771f-982e-4022-a1de-ab6373b7dab1-tls-certs" pod "00b8771f-982e-4022-a1de-ab6373b7dab1" (UID: "00b8771f-982e-4022-a1de-ab6373b7dab1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:53.015652 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.015618 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b8771f-982e-4022-a1de-ab6373b7dab1-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:53.015652 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.015647 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rbm\" (UniqueName: \"kubernetes.io/projected/00b8771f-982e-4022-a1de-ab6373b7dab1-kube-api-access-82rbm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:53.015652 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.015657 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:53.015980 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.015666 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b8771f-982e-4022-a1de-ab6373b7dab1-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:53.308090 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.307851 4477 generic.go:356] "Generic (PLEG): container finished" podID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerID="9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4" exitCode=0 Nov 01 21:15:53.308090 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.307910 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" Nov 01 21:15:53.308090 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.307937 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" event={"ID":"00b8771f-982e-4022-a1de-ab6373b7dab1","Type":"ContainerDied","Data":"9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4"} Nov 01 21:15:53.308090 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.307981 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc" event={"ID":"00b8771f-982e-4022-a1de-ab6373b7dab1","Type":"ContainerDied","Data":"b7dbeb3de658e19324383e9a114891f4518d70a8f7a293fe57eb817685a4c0f6"} Nov 01 21:15:53.308090 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.308008 4477 scope.go:122] "RemoveContainer" containerID="9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4" Nov 01 21:15:53.318918 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.318896 4477 scope.go:122] "RemoveContainer" containerID="9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4" Nov 01 21:15:53.319281 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:53.319247 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4\": container with ID starting with 9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4 not found: ID does not exist" containerID="9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4" Nov 01 21:15:53.319383 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.319286 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4"} err="failed to get container status \"9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4\": rpc error: code = NotFound desc = could not find container \"9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4\": container with ID starting with 9abee7d6507cee6c9d760eb106b57bf7381b2e8a8cd5865476041723f8f533e4 not found: ID does not exist" Nov 01 21:15:53.484713 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.484678 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc"] Nov 01 21:15:53.533818 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:53.533786 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/simulator-w0-208f2b-kserve-7bb675d57f-wt5fc"] Nov 01 21:15:54.503878 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:54.503843 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" path="/var/lib/kubelet/pods/00b8771f-982e-4022-a1de-ab6373b7dab1/volumes" Nov 01 21:15:57.835418 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:57.835387 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8"] Nov 01 21:15:57.835826 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:57.835577 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" podUID="ad6ef13d-442d-48e5-9a43-905aa13428a3" containerName="maas-api" containerID="cri-o://05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255" gracePeriod=30 Nov 01 21:15:58.158164 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.158134 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 21:15:58.271120 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.271066 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6ef13d-442d-48e5-9a43-905aa13428a3-kube-api-access-xvmfs\" (UniqueName: \"kubernetes.io/projected/ad6ef13d-442d-48e5-9a43-905aa13428a3-kube-api-access-xvmfs\") pod \"ad6ef13d-442d-48e5-9a43-905aa13428a3\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " Nov 01 21:15:58.271323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.271143 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6ef13d-442d-48e5-9a43-905aa13428a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6ef13d-442d-48e5-9a43-905aa13428a3-openshift-service-ca\") pod \"ad6ef13d-442d-48e5-9a43-905aa13428a3\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " Nov 01 21:15:58.271323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.271221 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls\") pod \"ad6ef13d-442d-48e5-9a43-905aa13428a3\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " Nov 01 21:15:58.271323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.271246 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-metrics-certs\") pod \"ad6ef13d-442d-48e5-9a43-905aa13428a3\" (UID: \"ad6ef13d-442d-48e5-9a43-905aa13428a3\") " Nov 01 21:15:58.271516 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.271471 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6ef13d-442d-48e5-9a43-905aa13428a3-openshift-service-ca" pod "ad6ef13d-442d-48e5-9a43-905aa13428a3" (UID: "ad6ef13d-442d-48e5-9a43-905aa13428a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:58.271710 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.271690 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6ef13d-442d-48e5-9a43-905aa13428a3-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:58.273623 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.273583 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls" pod "ad6ef13d-442d-48e5-9a43-905aa13428a3" (UID: "ad6ef13d-442d-48e5-9a43-905aa13428a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:58.274395 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.274109 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-metrics-certs" pod "ad6ef13d-442d-48e5-9a43-905aa13428a3" (UID: "ad6ef13d-442d-48e5-9a43-905aa13428a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:58.274395 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.274146 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6ef13d-442d-48e5-9a43-905aa13428a3-kube-api-access-xvmfs" pod "ad6ef13d-442d-48e5-9a43-905aa13428a3" (UID: "ad6ef13d-442d-48e5-9a43-905aa13428a3"). InnerVolumeSpecName "kube-api-access-xvmfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:58.341362 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.341323 4477 generic.go:356] "Generic (PLEG): container finished" podID="ad6ef13d-442d-48e5-9a43-905aa13428a3" containerID="05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255" exitCode=0 Nov 01 21:15:58.341539 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.341374 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" Nov 01 21:15:58.341539 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.341418 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" event={"ID":"ad6ef13d-442d-48e5-9a43-905aa13428a3","Type":"ContainerDied","Data":"05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255"} Nov 01 21:15:58.341539 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.341481 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8" event={"ID":"ad6ef13d-442d-48e5-9a43-905aa13428a3","Type":"ContainerDied","Data":"a199c5e6c204ebe5a672d0c729c884d7d1bc992dc47135086508738ac05c9d0f"} Nov 01 21:15:58.341704 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.341543 4477 scope.go:122] "RemoveContainer" containerID="05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255" Nov 01 21:15:58.352908 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.352888 4477 scope.go:122] "RemoveContainer" containerID="05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255" Nov 01 21:15:58.353197 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:15:58.353178 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255\": container with ID starting with 05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255 not found: ID does not exist" containerID="05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255" Nov 01 21:15:58.353251 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.353207 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255"} err="failed to get container status \"05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255\": rpc error: code = NotFound desc = could not find container \"05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255\": container with ID starting with 05d3ff148c290a23d0ffb44848a5d99ac894deae6622e62747b5c9b3f1dab255 not found: ID does not exist" Nov 01 21:15:58.372755 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.372730 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmfs\" (UniqueName: \"kubernetes.io/projected/ad6ef13d-442d-48e5-9a43-905aa13428a3-kube-api-access-xvmfs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:58.372755 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.372754 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:58.372905 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.372764 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6ef13d-442d-48e5-9a43-905aa13428a3-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:58.512207 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.512114 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8"] Nov 01 21:15:58.514315 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:15:58.514272 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-df700a-7744bb4674-l2wx8"] Nov 01 21:16:00.503639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:00.503608 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6ef13d-442d-48e5-9a43-905aa13428a3" path="/var/lib/kubelet/pods/ad6ef13d-442d-48e5-9a43-905aa13428a3/volumes" Nov 01 21:16:06.555061 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:16:06.555019 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:16:07.635833 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:07.635794 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:16:09.975985 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:09.975947 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x"] Nov 01 21:16:09.976514 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:09.976223 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" podUID="23c4a40e-579b-4957-abe7-83fd4f082509" containerName="maas-api" containerID="cri-o://e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2" gracePeriod=30 Nov 01 21:16:10.230864 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.230796 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 21:16:10.390116 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.390088 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c4a40e-579b-4957-abe7-83fd4f082509-kube-api-access-cw282\" (UniqueName: \"kubernetes.io/projected/23c4a40e-579b-4957-abe7-83fd4f082509-kube-api-access-cw282\") pod \"23c4a40e-579b-4957-abe7-83fd4f082509\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " Nov 01 21:16:10.390290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.390140 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-metrics-certs\") pod \"23c4a40e-579b-4957-abe7-83fd4f082509\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " Nov 01 21:16:10.390290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.390202 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls\") pod \"23c4a40e-579b-4957-abe7-83fd4f082509\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " Nov 01 21:16:10.390290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.390227 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c4a40e-579b-4957-abe7-83fd4f082509-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c4a40e-579b-4957-abe7-83fd4f082509-openshift-service-ca\") pod \"23c4a40e-579b-4957-abe7-83fd4f082509\" (UID: \"23c4a40e-579b-4957-abe7-83fd4f082509\") " Nov 01 21:16:10.390673 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.390649 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c4a40e-579b-4957-abe7-83fd4f082509-openshift-service-ca" pod "23c4a40e-579b-4957-abe7-83fd4f082509" (UID: "23c4a40e-579b-4957-abe7-83fd4f082509"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:10.392436 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.392412 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls" pod "23c4a40e-579b-4957-abe7-83fd4f082509" (UID: "23c4a40e-579b-4957-abe7-83fd4f082509"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:10.392524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.392475 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c4a40e-579b-4957-abe7-83fd4f082509-kube-api-access-cw282" pod "23c4a40e-579b-4957-abe7-83fd4f082509" (UID: "23c4a40e-579b-4957-abe7-83fd4f082509"). InnerVolumeSpecName "kube-api-access-cw282". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:10.392568 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.392527 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-metrics-certs" pod "23c4a40e-579b-4957-abe7-83fd4f082509" (UID: "23c4a40e-579b-4957-abe7-83fd4f082509"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:10.413593 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.413563 4477 generic.go:356] "Generic (PLEG): container finished" podID="23c4a40e-579b-4957-abe7-83fd4f082509" containerID="e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2" exitCode=0 Nov 01 21:16:10.413752 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.413617 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" Nov 01 21:16:10.413752 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.413655 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" event={"ID":"23c4a40e-579b-4957-abe7-83fd4f082509","Type":"ContainerDied","Data":"e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2"} Nov 01 21:16:10.413752 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.413714 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x" event={"ID":"23c4a40e-579b-4957-abe7-83fd4f082509","Type":"ContainerDied","Data":"73147aa5d22cfb2fd61f4523e10edd5baa9f10f6de068d9c1b95c302ba459495"} Nov 01 21:16:10.413752 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.413739 4477 scope.go:122] "RemoveContainer" containerID="e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2" Nov 01 21:16:10.423989 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.423973 4477 scope.go:122] "RemoveContainer" containerID="e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2" Nov 01 21:16:10.424296 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:16:10.424277 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2\": container with ID starting with e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2 not found: ID does not exist" containerID="e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2" Nov 01 21:16:10.424376 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.424304 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2"} err="failed to get container status \"e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2\": rpc error: code = NotFound desc = could not find container \"e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2\": container with ID starting with e5735db53d4503699b3dabee7a300d031b22e3d9d26e778538372a96eca44de2 not found: ID does not exist" Nov 01 21:16:10.491602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.491529 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw282\" (UniqueName: \"kubernetes.io/projected/23c4a40e-579b-4957-abe7-83fd4f082509-kube-api-access-cw282\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:10.491602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.491557 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:10.491602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.491567 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c4a40e-579b-4957-abe7-83fd4f082509-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:10.491602 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.491577 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c4a40e-579b-4957-abe7-83fd4f082509-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:10.579896 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.579862 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x"] Nov 01 21:16:10.584154 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:10.584125 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf3fa2-668ccbb684-5wn4x"] Nov 01 21:16:12.278226 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:12.278192 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48"] Nov 01 21:16:12.327592 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:12.327555 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp"] Nov 01 21:16:12.507172 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:12.507132 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c4a40e-579b-4957-abe7-83fd4f082509" path="/var/lib/kubelet/pods/23c4a40e-579b-4957-abe7-83fd4f082509/volumes" Nov 01 21:16:16.536728 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.536684 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw"] Nov 01 21:16:16.537397 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.536911 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" podUID="68baa7a8-a64f-4f0b-84cc-47bb018042c7" containerName="maas-api" containerID="cri-o://abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162" gracePeriod=30 Nov 01 21:16:16.800471 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.800403 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 21:16:16.958572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.958499 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68baa7a8-a64f-4f0b-84cc-47bb018042c7-kube-api-access-qgzzc\" (UniqueName: \"kubernetes.io/projected/68baa7a8-a64f-4f0b-84cc-47bb018042c7-kube-api-access-qgzzc\") pod \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " Nov 01 21:16:16.958572 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.958566 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls\") pod \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " Nov 01 21:16:16.958814 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.958599 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68baa7a8-a64f-4f0b-84cc-47bb018042c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68baa7a8-a64f-4f0b-84cc-47bb018042c7-openshift-service-ca\") pod \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " Nov 01 21:16:16.958814 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.958661 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-metrics-certs\") pod \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\" (UID: \"68baa7a8-a64f-4f0b-84cc-47bb018042c7\") " Nov 01 21:16:16.960645 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.959370 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68baa7a8-a64f-4f0b-84cc-47bb018042c7-openshift-service-ca" pod "68baa7a8-a64f-4f0b-84cc-47bb018042c7" (UID: "68baa7a8-a64f-4f0b-84cc-47bb018042c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:16.961250 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.961224 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68baa7a8-a64f-4f0b-84cc-47bb018042c7-kube-api-access-qgzzc" pod "68baa7a8-a64f-4f0b-84cc-47bb018042c7" (UID: "68baa7a8-a64f-4f0b-84cc-47bb018042c7"). InnerVolumeSpecName "kube-api-access-qgzzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:16.961474 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.961448 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-metrics-certs" pod "68baa7a8-a64f-4f0b-84cc-47bb018042c7" (UID: "68baa7a8-a64f-4f0b-84cc-47bb018042c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:16.962177 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:16.962158 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls" pod "68baa7a8-a64f-4f0b-84cc-47bb018042c7" (UID: "68baa7a8-a64f-4f0b-84cc-47bb018042c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:17.060019 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.059928 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:17.060019 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.059965 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgzzc\" (UniqueName: \"kubernetes.io/projected/68baa7a8-a64f-4f0b-84cc-47bb018042c7-kube-api-access-qgzzc\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:17.060019 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.059981 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68baa7a8-a64f-4f0b-84cc-47bb018042c7-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:17.060019 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.059994 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68baa7a8-a64f-4f0b-84cc-47bb018042c7-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:17.458564 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.458525 4477 generic.go:356] "Generic (PLEG): container finished" podID="68baa7a8-a64f-4f0b-84cc-47bb018042c7" containerID="abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162" exitCode=0 Nov 01 21:16:17.458782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.458585 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" Nov 01 21:16:17.458782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.458611 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" event={"ID":"68baa7a8-a64f-4f0b-84cc-47bb018042c7","Type":"ContainerDied","Data":"abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162"} Nov 01 21:16:17.458782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.458697 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw" event={"ID":"68baa7a8-a64f-4f0b-84cc-47bb018042c7","Type":"ContainerDied","Data":"8a79f4497b9b4978120fd6e265f5488869144fdbaa8bddc12b6c86cf358ab9df"} Nov 01 21:16:17.458782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.458720 4477 scope.go:122] "RemoveContainer" containerID="abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162" Nov 01 21:16:17.469211 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.469194 4477 scope.go:122] "RemoveContainer" containerID="abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162" Nov 01 21:16:17.469504 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:16:17.469481 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162\": container with ID starting with abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162 not found: ID does not exist" containerID="abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162" Nov 01 21:16:17.469607 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.469514 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162"} err="failed to get container status \"abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162\": rpc error: code = NotFound desc = could not find container \"abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162\": container with ID starting with abb8445a69281ee55c7398b34c0c6113fea3a5f71521ffd1634f66bab34a7162 not found: ID does not exist" Nov 01 21:16:17.621588 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.621552 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw"] Nov 01 21:16:17.623623 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:17.623598 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1a1e9e-bb7c44fff-52dlw"] Nov 01 21:16:18.503693 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:18.503662 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68baa7a8-a64f-4f0b-84cc-47bb018042c7" path="/var/lib/kubelet/pods/68baa7a8-a64f-4f0b-84cc-47bb018042c7/volumes" Nov 01 21:16:24.150289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:24.150252 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97"] Nov 01 21:16:27.299269 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.299176 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" containerID="cri-o://697ef9e510b42fe7c82800cc7899e487ea8b6cb28f5ce307044a9c8aa6202175" gracePeriod=45 Nov 01 21:16:27.347607 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.347566 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" containerID="cri-o://095fd399398915640c9bac96ee5792104f369b3274e50164a7f713122ea2c8c5" gracePeriod=45 Nov 01 21:16:27.527993 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.527958 4477 generic.go:356] "Generic (PLEG): container finished" podID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerID="095fd399398915640c9bac96ee5792104f369b3274e50164a7f713122ea2c8c5" exitCode=0 Nov 01 21:16:27.528191 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.528172 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" event={"ID":"d6ec5583-c611-4b43-9233-b5bd641d801d","Type":"ContainerDied","Data":"095fd399398915640c9bac96ee5792104f369b3274e50164a7f713122ea2c8c5"} Nov 01 21:16:27.529990 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.529965 4477 generic.go:356] "Generic (PLEG): container finished" podID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerID="697ef9e510b42fe7c82800cc7899e487ea8b6cb28f5ce307044a9c8aa6202175" exitCode=0 Nov 01 21:16:27.530162 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.530023 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" event={"ID":"3abd0466-122f-449d-a2e9-82d6f408ef11","Type":"ContainerDied","Data":"697ef9e510b42fe7c82800cc7899e487ea8b6cb28f5ce307044a9c8aa6202175"} Nov 01 21:16:27.598175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.598148 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 21:16:27.625970 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.625946 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 21:16:27.645126 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645085 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ec5583-c611-4b43-9233-b5bd641d801d-kube-api-access-4rsnz\" (UniqueName: \"kubernetes.io/projected/d6ec5583-c611-4b43-9233-b5bd641d801d-kube-api-access-4rsnz\") pod \"d6ec5583-c611-4b43-9233-b5bd641d801d\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " Nov 01 21:16:27.645253 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645131 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-tmp-dir\") pod \"3abd0466-122f-449d-a2e9-82d6f408ef11\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " Nov 01 21:16:27.645253 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645158 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-model-cache\") pod \"d6ec5583-c611-4b43-9233-b5bd641d801d\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " Nov 01 21:16:27.645253 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645188 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3abd0466-122f-449d-a2e9-82d6f408ef11-tls-certs\" (UniqueName: \"kubernetes.io/secret/3abd0466-122f-449d-a2e9-82d6f408ef11-tls-certs\") pod \"3abd0466-122f-449d-a2e9-82d6f408ef11\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " Nov 01 21:16:27.645397 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645262 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-model-cache\") pod \"3abd0466-122f-449d-a2e9-82d6f408ef11\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " Nov 01 21:16:27.645397 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645290 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-dshm\") pod \"3abd0466-122f-449d-a2e9-82d6f408ef11\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " Nov 01 21:16:27.645397 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645331 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-home\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-home\") pod \"d6ec5583-c611-4b43-9233-b5bd641d801d\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " Nov 01 21:16:27.645397 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645377 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abd0466-122f-449d-a2e9-82d6f408ef11-kube-api-access-672pv\" (UniqueName: \"kubernetes.io/projected/3abd0466-122f-449d-a2e9-82d6f408ef11-kube-api-access-672pv\") pod \"3abd0466-122f-449d-a2e9-82d6f408ef11\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " Nov 01 21:16:27.645617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645404 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-tmp-dir\") pod \"d6ec5583-c611-4b43-9233-b5bd641d801d\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " Nov 01 21:16:27.645617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645406 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-model-cache" pod "d6ec5583-c611-4b43-9233-b5bd641d801d" (UID: "d6ec5583-c611-4b43-9233-b5bd641d801d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.645617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645396 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-tmp-dir" pod "3abd0466-122f-449d-a2e9-82d6f408ef11" (UID: "3abd0466-122f-449d-a2e9-82d6f408ef11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.645617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645452 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-home\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-home\") pod \"3abd0466-122f-449d-a2e9-82d6f408ef11\" (UID: \"3abd0466-122f-449d-a2e9-82d6f408ef11\") " Nov 01 21:16:27.645617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645488 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6ec5583-c611-4b43-9233-b5bd641d801d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ec5583-c611-4b43-9233-b5bd641d801d-tls-certs\") pod \"d6ec5583-c611-4b43-9233-b5bd641d801d\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " Nov 01 21:16:27.645617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645530 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-dshm\") pod \"d6ec5583-c611-4b43-9233-b5bd641d801d\" (UID: \"d6ec5583-c611-4b43-9233-b5bd641d801d\") " Nov 01 21:16:27.645617 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645571 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-model-cache" pod "3abd0466-122f-449d-a2e9-82d6f408ef11" (UID: "3abd0466-122f-449d-a2e9-82d6f408ef11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.645934 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645676 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-home" pod "d6ec5583-c611-4b43-9233-b5bd641d801d" (UID: "d6ec5583-c611-4b43-9233-b5bd641d801d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.645934 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645732 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-tmp-dir" pod "d6ec5583-c611-4b43-9233-b5bd641d801d" (UID: "d6ec5583-c611-4b43-9233-b5bd641d801d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.645934 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645891 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.645934 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645912 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.645934 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645928 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.646231 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645942 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.646231 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.645954 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.646231 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.646210 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-home" pod "3abd0466-122f-449d-a2e9-82d6f408ef11" (UID: "3abd0466-122f-449d-a2e9-82d6f408ef11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.648142 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.648113 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-dshm" pod "d6ec5583-c611-4b43-9233-b5bd641d801d" (UID: "d6ec5583-c611-4b43-9233-b5bd641d801d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.648316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.648177 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-dshm" pod "3abd0466-122f-449d-a2e9-82d6f408ef11" (UID: "3abd0466-122f-449d-a2e9-82d6f408ef11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.648316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.648266 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abd0466-122f-449d-a2e9-82d6f408ef11-kube-api-access-672pv" pod "3abd0466-122f-449d-a2e9-82d6f408ef11" (UID: "3abd0466-122f-449d-a2e9-82d6f408ef11"). InnerVolumeSpecName "kube-api-access-672pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:27.648316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.648296 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ec5583-c611-4b43-9233-b5bd641d801d-tls-certs" pod "d6ec5583-c611-4b43-9233-b5bd641d801d" (UID: "d6ec5583-c611-4b43-9233-b5bd641d801d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:27.648316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.648305 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3abd0466-122f-449d-a2e9-82d6f408ef11-tls-certs" pod "3abd0466-122f-449d-a2e9-82d6f408ef11" (UID: "3abd0466-122f-449d-a2e9-82d6f408ef11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:27.648468 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.648324 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ec5583-c611-4b43-9233-b5bd641d801d-kube-api-access-4rsnz" pod "d6ec5583-c611-4b43-9233-b5bd641d801d" (UID: "d6ec5583-c611-4b43-9233-b5bd641d801d"). InnerVolumeSpecName "kube-api-access-4rsnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:27.746758 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.746723 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672pv\" (UniqueName: \"kubernetes.io/projected/3abd0466-122f-449d-a2e9-82d6f408ef11-kube-api-access-672pv\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.746758 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.746751 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.746758 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.746761 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ec5583-c611-4b43-9233-b5bd641d801d-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.746758 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.746769 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ec5583-c611-4b43-9233-b5bd641d801d-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.747212 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.746778 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rsnz\" (UniqueName: \"kubernetes.io/projected/d6ec5583-c611-4b43-9233-b5bd641d801d-kube-api-access-4rsnz\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.747212 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.746787 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abd0466-122f-449d-a2e9-82d6f408ef11-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.747212 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:27.746794 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abd0466-122f-449d-a2e9-82d6f408ef11-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:28.537107 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.537044 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" Nov 01 21:16:28.537660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.537121 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48" event={"ID":"3abd0466-122f-449d-a2e9-82d6f408ef11","Type":"ContainerDied","Data":"c4b52518766d96cf98538deeb211411ec233c73dbf7b20630be45445ee5b735c"} Nov 01 21:16:28.537660 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.537177 4477 scope.go:122] "RemoveContainer" containerID="697ef9e510b42fe7c82800cc7899e487ea8b6cb28f5ce307044a9c8aa6202175" Nov 01 21:16:28.539368 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.539335 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" event={"ID":"d6ec5583-c611-4b43-9233-b5bd641d801d","Type":"ContainerDied","Data":"a0d274040972de8f76d399db170cf87cef5b04e996472444115b48d5f9dc87b7"} Nov 01 21:16:28.539494 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.539391 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp" Nov 01 21:16:28.548099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.548046 4477 scope.go:122] "RemoveContainer" containerID="095fd399398915640c9bac96ee5792104f369b3274e50164a7f713122ea2c8c5" Nov 01 21:16:28.699802 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.699702 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp"] Nov 01 21:16:28.760753 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.760701 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/premium-w1-db0157-kserve-75fb4d65cd-r7zcp"] Nov 01 21:16:28.811003 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.810961 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48"] Nov 01 21:16:28.817713 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:28.817679 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-2-w1-db0157-kserve-67c894bb6-8rv48"] Nov 01 21:16:30.502922 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:30.502883 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" path="/var/lib/kubelet/pods/3abd0466-122f-449d-a2e9-82d6f408ef11/volumes" Nov 01 21:16:30.503367 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:30.503300 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" path="/var/lib/kubelet/pods/d6ec5583-c611-4b43-9233-b5bd641d801d/volumes" Nov 01 21:16:33.552814 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:33.552774 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:16:34.159966 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.159931 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm"] Nov 01 21:16:34.160382 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160360 4477 cpu_manager.go:414] "containerMap: removing container" podUID="68baa7a8-a64f-4f0b-84cc-47bb018042c7" containerName="maas-api" Nov 01 21:16:34.160382 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160379 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68baa7a8-a64f-4f0b-84cc-47bb018042c7" containerName="maas-api" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160390 4477 cpu_manager.go:414] "containerMap: removing container" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160395 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160403 4477 cpu_manager.go:414] "containerMap: removing container" podUID="8578e043-d4a0-4968-a7f7-b9fa052c8fb1" containerName="maas-api" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160410 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8578e043-d4a0-4968-a7f7-b9fa052c8fb1" containerName="maas-api" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160427 4477 cpu_manager.go:414] "containerMap: removing container" podUID="23c4a40e-579b-4957-abe7-83fd4f082509" containerName="maas-api" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160434 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c4a40e-579b-4957-abe7-83fd4f082509" containerName="maas-api" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160466 4477 cpu_manager.go:414] "containerMap: removing container" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160472 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160481 4477 cpu_manager.go:414] "containerMap: removing container" podUID="ad6ef13d-442d-48e5-9a43-905aa13428a3" containerName="maas-api" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160489 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6ef13d-442d-48e5-9a43-905aa13428a3" containerName="maas-api" Nov 01 21:16:34.160614 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160590 4477 cpu_manager.go:414] "containerMap: removing container" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160621 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160634 4477 cpu_manager.go:414] "containerMap: removing container" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160639 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160645 4477 cpu_manager.go:414] "containerMap: removing container" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160650 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160660 4477 cpu_manager.go:414] "containerMap: removing container" podUID="bb31a6c3-1a59-4b34-b906-d1a335333566" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160664 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb31a6c3-1a59-4b34-b906-d1a335333566" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160671 4477 cpu_manager.go:414] "containerMap: removing container" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160676 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160681 4477 cpu_manager.go:414] "containerMap: removing container" podUID="fcb2250c-fc1b-4982-82d5-678fb31f35f3" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160687 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb2250c-fc1b-4982-82d5-678fb31f35f3" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160701 4477 cpu_manager.go:414] "containerMap: removing container" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160705 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160719 4477 cpu_manager.go:414] "containerMap: removing container" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160727 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160847 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160859 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6ef13d-442d-48e5-9a43-905aa13428a3" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160871 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd279d92-f03e-4a2e-8a0d-b785efaeace6" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160879 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160893 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b8771f-982e-4022-a1de-ab6373b7dab1" containerName="main" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160903 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb31a6c3-1a59-4b34-b906-d1a335333566" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160912 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f591f6-2c37-47df-b609-08d801eb30fa" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160919 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb2250c-fc1b-4982-82d5-678fb31f35f3" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160928 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ec5583-c611-4b43-9233-b5bd641d801d" containerName="main" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160937 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f054e8-f0c0-4968-b8b2-c76fafb1876d" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160949 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dea6272-9dbc-435f-a85d-721b92bc7302" containerName="istio-proxy" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160959 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="68baa7a8-a64f-4f0b-84cc-47bb018042c7" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160968 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abd0466-122f-449d-a2e9-82d6f408ef11" containerName="main" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160977 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8578e043-d4a0-4968-a7f7-b9fa052c8fb1" containerName="maas-api" Nov 01 21:16:34.161175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.160985 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c4a40e-579b-4957-abe7-83fd4f082509" containerName="maas-api" Nov 01 21:16:34.165726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.165709 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.211404 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.211384 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.211524 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.211457 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krtd\" (UniqueName: \"kubernetes.io/projected/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-kube-api-access-4krtd\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.211577 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.211559 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.219640 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.219614 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm"] Nov 01 21:16:34.266507 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.266483 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:16:34.266600 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.266518 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:16:34.312684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.312663 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.312799 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.312729 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krtd\" (UniqueName: \"kubernetes.io/projected/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-kube-api-access-4krtd\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.312799 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.312752 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.313358 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.313341 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.315170 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.315154 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.369724 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.369691 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krtd\" (UniqueName: \"kubernetes.io/projected/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-kube-api-access-4krtd\") pod \"maas-api-revoke-sub-78bba8d8-dthwm\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.478828 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.478750 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:34.858132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:34.858105 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm"] Nov 01 21:16:34.860545 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:16:34.860516 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2da6f1_8b9c_45c0_8b46_0abcc45f422a.slice/crio-9d84bc4779362d0ccd07ec0e3e3a966a5812bc6a2c4e9039f95ea0a5db643eec WatchSource:0}: Error finding container 9d84bc4779362d0ccd07ec0e3e3a966a5812bc6a2c4e9039f95ea0a5db643eec: Status 404 returned error can't find the container with id 9d84bc4779362d0ccd07ec0e3e3a966a5812bc6a2c4e9039f95ea0a5db643eec Nov 01 21:16:35.580470 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:35.580438 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" event={"ID":"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a","Type":"ContainerStarted","Data":"fc41c40aed2e677e929df89f07d615a65f0ecaeb939a3cce5c45aaffc99a2cba"} Nov 01 21:16:35.580669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:35.580488 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" event={"ID":"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a","Type":"ContainerStarted","Data":"9d84bc4779362d0ccd07ec0e3e3a966a5812bc6a2c4e9039f95ea0a5db643eec"} Nov 01 21:16:35.694902 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:35.694848 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" podStartSLOduration=1.694833342 podStartE2EDuration="1.694833342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:35.692185835 +0000 UTC m=+2749.342237404" watchObservedRunningTime="2026-11-01 21:16:35.694833342 +0000 UTC m=+2749.344884886" Nov 01 21:16:36.553434 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:16:36.553393 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:16:36.587125 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:36.587068 4477 generic.go:356] "Generic (PLEG): container finished" podID="2e2da6f1-8b9c-45c0-8b46-0abcc45f422a" containerID="fc41c40aed2e677e929df89f07d615a65f0ecaeb939a3cce5c45aaffc99a2cba" exitCode=0 Nov 01 21:16:36.587317 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:36.587135 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" event={"ID":"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a","Type":"ContainerDied","Data":"fc41c40aed2e677e929df89f07d615a65f0ecaeb939a3cce5c45aaffc99a2cba"} Nov 01 21:16:37.731006 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.730979 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:37.845891 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.845800 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-xks-ca\") pod \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " Nov 01 21:16:37.845891 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.845866 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-openshift-ca\") pod \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " Nov 01 21:16:37.846159 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.845914 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-kube-api-access-4krtd\" (UniqueName: \"kubernetes.io/projected/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-kube-api-access-4krtd\") pod \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\" (UID: \"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a\") " Nov 01 21:16:37.846328 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.846301 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-openshift-ca" pod "2e2da6f1-8b9c-45c0-8b46-0abcc45f422a" (UID: "2e2da6f1-8b9c-45c0-8b46-0abcc45f422a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:37.848186 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.848155 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-xks-ca" pod "2e2da6f1-8b9c-45c0-8b46-0abcc45f422a" (UID: "2e2da6f1-8b9c-45c0-8b46-0abcc45f422a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:37.848362 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.848342 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-kube-api-access-4krtd" pod "2e2da6f1-8b9c-45c0-8b46-0abcc45f422a" (UID: "2e2da6f1-8b9c-45c0-8b46-0abcc45f422a"). InnerVolumeSpecName "kube-api-access-4krtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:37.946737 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.946700 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-openshift-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:37.946737 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.946730 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krtd\" (UniqueName: \"kubernetes.io/projected/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-kube-api-access-4krtd\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:37.946737 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:37.946740 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a-maas-api-xks-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:38.599432 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:38.599400 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" Nov 01 21:16:38.599622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:38.599440 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm" event={"ID":"2e2da6f1-8b9c-45c0-8b46-0abcc45f422a","Type":"ContainerDied","Data":"9d84bc4779362d0ccd07ec0e3e3a966a5812bc6a2c4e9039f95ea0a5db643eec"} Nov 01 21:16:38.599622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:38.599489 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d84bc4779362d0ccd07ec0e3e3a966a5812bc6a2c4e9039f95ea0a5db643eec" Nov 01 21:16:39.170290 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.170255 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr"] Nov 01 21:16:39.170687 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.170655 4477 cpu_manager.go:414] "containerMap: removing container" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:39.170687 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.170669 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:39.170763 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.170688 4477 cpu_manager.go:414] "containerMap: removing container" podUID="2e2da6f1-8b9c-45c0-8b46-0abcc45f422a" containerName="revoke-keys" Nov 01 21:16:39.170763 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.170696 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2da6f1-8b9c-45c0-8b46-0abcc45f422a" containerName="revoke-keys" Nov 01 21:16:39.170763 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.170759 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="872a3054-33bc-41ab-9f16-903afc70de64" containerName="cleanup" Nov 01 21:16:39.170901 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.170775 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2da6f1-8b9c-45c0-8b46-0abcc45f422a" containerName="revoke-keys" Nov 01 21:16:39.172592 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.172551 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" containerID="cri-o://8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d" gracePeriod=45 Nov 01 21:16:39.173792 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.173774 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.229107 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.229065 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr"] Nov 01 21:16:39.260664 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.260638 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nlld\" (UniqueName: \"kubernetes.io/projected/96e51240-447c-4b20-b2d1-9e5611e9ae67-kube-api-access-6nlld\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.260782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.260766 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.260846 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.260808 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.268796 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.268777 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:16:39.268900 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.268856 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:16:39.361655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.361608 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.361844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.361681 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlld\" (UniqueName: \"kubernetes.io/projected/96e51240-447c-4b20-b2d1-9e5611e9ae67-kube-api-access-6nlld\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.361844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.361777 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.362391 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.362358 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.364441 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.364422 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.425433 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.425366 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nlld\" (UniqueName: \"kubernetes.io/projected/96e51240-447c-4b20-b2d1-9e5611e9ae67-kube-api-access-6nlld\") pod \"maas-api-revoke-sub-2c8c3637-xtjdr\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.433923 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.433898 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 21:16:39.462898 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.462867 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-dshm\") pod \"5914cef2-d12a-4875-b1d2-02ead983c528\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " Nov 01 21:16:39.463067 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.462927 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-home\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-home\") pod \"5914cef2-d12a-4875-b1d2-02ead983c528\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " Nov 01 21:16:39.463067 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.462978 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5914cef2-d12a-4875-b1d2-02ead983c528-tls-certs\" (UniqueName: \"kubernetes.io/secret/5914cef2-d12a-4875-b1d2-02ead983c528-tls-certs\") pod \"5914cef2-d12a-4875-b1d2-02ead983c528\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " Nov 01 21:16:39.463067 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.463000 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5914cef2-d12a-4875-b1d2-02ead983c528-kube-api-access-knzg6\" (UniqueName: \"kubernetes.io/projected/5914cef2-d12a-4875-b1d2-02ead983c528-kube-api-access-knzg6\") pod \"5914cef2-d12a-4875-b1d2-02ead983c528\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " Nov 01 21:16:39.463067 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.463044 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-model-cache\") pod \"5914cef2-d12a-4875-b1d2-02ead983c528\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " Nov 01 21:16:39.463337 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.463120 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-tmp-dir\") pod \"5914cef2-d12a-4875-b1d2-02ead983c528\" (UID: \"5914cef2-d12a-4875-b1d2-02ead983c528\") " Nov 01 21:16:39.463337 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.463279 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-home" pod "5914cef2-d12a-4875-b1d2-02ead983c528" (UID: "5914cef2-d12a-4875-b1d2-02ead983c528"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:39.463435 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.463357 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-model-cache" pod "5914cef2-d12a-4875-b1d2-02ead983c528" (UID: "5914cef2-d12a-4875-b1d2-02ead983c528"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:39.463489 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.463426 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-tmp-dir" pod "5914cef2-d12a-4875-b1d2-02ead983c528" (UID: "5914cef2-d12a-4875-b1d2-02ead983c528"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:39.463549 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.463498 4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-home\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.463549 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.463520 4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-model-cache\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.465273 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.465244 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-dshm" pod "5914cef2-d12a-4875-b1d2-02ead983c528" (UID: "5914cef2-d12a-4875-b1d2-02ead983c528"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:39.465380 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.465306 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5914cef2-d12a-4875-b1d2-02ead983c528-tls-certs" pod "5914cef2-d12a-4875-b1d2-02ead983c528" (UID: "5914cef2-d12a-4875-b1d2-02ead983c528"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:39.465380 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.465310 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5914cef2-d12a-4875-b1d2-02ead983c528-kube-api-access-knzg6" pod "5914cef2-d12a-4875-b1d2-02ead983c528" (UID: "5914cef2-d12a-4875-b1d2-02ead983c528"). InnerVolumeSpecName "kube-api-access-knzg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:39.565781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.565596 4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-tmp-dir\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.565781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.565692 4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5914cef2-d12a-4875-b1d2-02ead983c528-dshm\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.565781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.565723 4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5914cef2-d12a-4875-b1d2-02ead983c528-tls-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.565781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.565755 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knzg6\" (UniqueName: \"kubernetes.io/projected/5914cef2-d12a-4875-b1d2-02ead983c528-kube-api-access-knzg6\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.565781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.565755 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:39.609860 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.609820 4477 generic.go:356] "Generic (PLEG): container finished" podID="5914cef2-d12a-4875-b1d2-02ead983c528" containerID="8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d" exitCode=0 Nov 01 21:16:39.610066 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.609874 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" event={"ID":"5914cef2-d12a-4875-b1d2-02ead983c528","Type":"ContainerDied","Data":"8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d"} Nov 01 21:16:39.610066 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.609910 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" event={"ID":"5914cef2-d12a-4875-b1d2-02ead983c528","Type":"ContainerDied","Data":"f456d617deb6e85210c8832cb08ff5b1db2e7ffbeed399946b9630ea10d3188f"} Nov 01 21:16:39.610066 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.609933 4477 scope.go:122] "RemoveContainer" containerID="8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d" Nov 01 21:16:39.610267 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.610124 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97" Nov 01 21:16:39.622278 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.622254 4477 scope.go:122] "RemoveContainer" containerID="8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d" Nov 01 21:16:39.622645 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:16:39.622622 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d\": container with ID starting with 8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d not found: ID does not exist" containerID="8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d" Nov 01 21:16:39.622749 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.622657 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d"} err="failed to get container status \"8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d\": rpc error: code = NotFound desc = could not find container \"8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d\": container with ID starting with 8aa55b8dd791694846e490552f3db8129c985a6fe0386578e412c9325eee666d not found: ID does not exist" Nov 01 21:16:39.774067 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.773964 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97"] Nov 01 21:16:39.779842 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.779816 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-w3-944263-kserve-55d477bd74-97p97"] Nov 01 21:16:39.952819 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:39.952797 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr"] Nov 01 21:16:39.954710 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:16:39.954680 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e51240_447c_4b20_b2d1_9e5611e9ae67.slice/crio-2c2fca5f96554eda58d6873115e55eeeec8f1d4857237ac8c29e1c572d082f6f WatchSource:0}: Error finding container 2c2fca5f96554eda58d6873115e55eeeec8f1d4857237ac8c29e1c572d082f6f: Status 404 returned error can't find the container with id 2c2fca5f96554eda58d6873115e55eeeec8f1d4857237ac8c29e1c572d082f6f Nov 01 21:16:40.502589 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:40.502514 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" path="/var/lib/kubelet/pods/5914cef2-d12a-4875-b1d2-02ead983c528/volumes" Nov 01 21:16:40.628236 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:40.628190 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" event={"ID":"96e51240-447c-4b20-b2d1-9e5611e9ae67","Type":"ContainerStarted","Data":"167ff944a699d9d9d3c804bc9474aa61325ba5520984eb4873c62c337ad31a4d"} Nov 01 21:16:40.628634 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:40.628291 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" event={"ID":"96e51240-447c-4b20-b2d1-9e5611e9ae67","Type":"ContainerStarted","Data":"2c2fca5f96554eda58d6873115e55eeeec8f1d4857237ac8c29e1c572d082f6f"} Nov 01 21:16:40.741615 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:40.741563 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" podStartSLOduration=1.741546694 podStartE2EDuration="1.741546694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:40.739773644 +0000 UTC m=+2754.389825190" watchObservedRunningTime="2026-11-01 21:16:40.741546694 +0000 UTC m=+2754.391598239" Nov 01 21:16:41.638336 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:41.638295 4477 generic.go:356] "Generic (PLEG): container finished" podID="96e51240-447c-4b20-b2d1-9e5611e9ae67" containerID="167ff944a699d9d9d3c804bc9474aa61325ba5520984eb4873c62c337ad31a4d" exitCode=0 Nov 01 21:16:41.638769 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:41.638343 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" event={"ID":"96e51240-447c-4b20-b2d1-9e5611e9ae67","Type":"ContainerDied","Data":"167ff944a699d9d9d3c804bc9474aa61325ba5520984eb4873c62c337ad31a4d"} Nov 01 21:16:42.785873 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.785841 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:42.898181 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.897943 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-xks-ca\") pod \"96e51240-447c-4b20-b2d1-9e5611e9ae67\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " Nov 01 21:16:42.898181 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.898014 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e51240-447c-4b20-b2d1-9e5611e9ae67-kube-api-access-6nlld\" (UniqueName: \"kubernetes.io/projected/96e51240-447c-4b20-b2d1-9e5611e9ae67-kube-api-access-6nlld\") pod \"96e51240-447c-4b20-b2d1-9e5611e9ae67\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " Nov 01 21:16:42.898181 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.898136 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-openshift-ca\") pod \"96e51240-447c-4b20-b2d1-9e5611e9ae67\" (UID: \"96e51240-447c-4b20-b2d1-9e5611e9ae67\") " Nov 01 21:16:42.898822 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.898790 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-openshift-ca" pod "96e51240-447c-4b20-b2d1-9e5611e9ae67" (UID: "96e51240-447c-4b20-b2d1-9e5611e9ae67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:42.900945 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.900916 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e51240-447c-4b20-b2d1-9e5611e9ae67-kube-api-access-6nlld" pod "96e51240-447c-4b20-b2d1-9e5611e9ae67" (UID: "96e51240-447c-4b20-b2d1-9e5611e9ae67"). InnerVolumeSpecName "kube-api-access-6nlld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:42.901142 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.901117 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-xks-ca" pod "96e51240-447c-4b20-b2d1-9e5611e9ae67" (UID: "96e51240-447c-4b20-b2d1-9e5611e9ae67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:42.999645 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.999611 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-xks-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:42.999645 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.999638 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nlld\" (UniqueName: \"kubernetes.io/projected/96e51240-447c-4b20-b2d1-9e5611e9ae67-kube-api-access-6nlld\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:42.999645 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:42.999649 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e51240-447c-4b20-b2d1-9e5611e9ae67-maas-api-openshift-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:43.652353 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:43.652313 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" event={"ID":"96e51240-447c-4b20-b2d1-9e5611e9ae67","Type":"ContainerDied","Data":"2c2fca5f96554eda58d6873115e55eeeec8f1d4857237ac8c29e1c572d082f6f"} Nov 01 21:16:43.652543 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:43.652355 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr" Nov 01 21:16:43.652543 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:43.652362 4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c2fca5f96554eda58d6873115e55eeeec8f1d4857237ac8c29e1c572d082f6f" Nov 01 21:16:55.955918 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:16:55.955878 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:17:06.552571 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:17:06.552534 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:17:33.120031 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.119985 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6"] Nov 01 21:17:33.120727 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.120583 4477 cpu_manager.go:414] "containerMap: removing container" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" Nov 01 21:17:33.120727 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.120603 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" Nov 01 21:17:33.120727 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.120633 4477 cpu_manager.go:414] "containerMap: removing container" podUID="96e51240-447c-4b20-b2d1-9e5611e9ae67" containerName="revoke-keys" Nov 01 21:17:33.120727 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.120642 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e51240-447c-4b20-b2d1-9e5611e9ae67" containerName="revoke-keys" Nov 01 21:17:33.120957 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.120757 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="5914cef2-d12a-4875-b1d2-02ead983c528" containerName="main" Nov 01 21:17:33.120957 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.120771 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e51240-447c-4b20-b2d1-9e5611e9ae67" containerName="revoke-keys" Nov 01 21:17:33.124044 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.124024 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224134 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224099 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224148 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224231 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224312 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224289 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224505 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224384 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224505 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224446 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224505 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224497 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fw28\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-kube-api-access-4fw28\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224671 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224538 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a515b52-f936-4e88-ad17-a49ef4c2451c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.224671 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.224646 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.325782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.325739 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.326535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.325787 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fw28\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-kube-api-access-4fw28\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.326535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.325824 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a515b52-f936-4e88-ad17-a49ef4c2451c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.326535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.325880 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.326535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.325930 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.326535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.325957 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.326535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.325985 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.326535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.326214 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.326535 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.326246 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.327528 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.326827 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.327528 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.327236 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.327528 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.327508 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.327817 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.327794 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a515b52-f936-4e88-ad17-a49ef4c2451c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.328220 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.328195 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.328650 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.328498 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6"] Nov 01 21:17:33.331243 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.331216 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.331376 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.331223 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.382149 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.382057 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fw28\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-kube-api-access-4fw28\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.382149 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.382061 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bb8j6\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.437228 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.437189 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:33.866637 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.866589 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6"] Nov 01 21:17:33.868238 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:17:33.868209 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a515b52_f936_4e88_ad17_a49ef4c2451c.slice/crio-dd1a95ff316b496ad2c0d384350b2408be3d6eadc6a3bf188c6af5c5dc4c9006 WatchSource:0}: Error finding container dd1a95ff316b496ad2c0d384350b2408be3d6eadc6a3bf188c6af5c5dc4c9006: Status 404 returned error can't find the container with id dd1a95ff316b496ad2c0d384350b2408be3d6eadc6a3bf188c6af5c5dc4c9006 Nov 01 21:17:33.870852 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.870817 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:17:33.870961 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.870904 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:17:33.870961 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.870949 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:17:33.931129 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:33.931099 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" event={"ID":"0a515b52-f936-4e88-ad17-a49ef4c2451c","Type":"ContainerStarted","Data":"dd1a95ff316b496ad2c0d384350b2408be3d6eadc6a3bf188c6af5c5dc4c9006"} Nov 01 21:17:34.942643 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:34.942607 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" event={"ID":"0a515b52-f936-4e88-ad17-a49ef4c2451c","Type":"ContainerStarted","Data":"641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167"} Nov 01 21:17:35.107638 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:35.107579 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" podStartSLOduration=2.107561862 podStartE2EDuration="2.107561862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:35.05662123 +0000 UTC m=+2808.706672790" watchObservedRunningTime="2026-11-01 21:17:35.107561862 +0000 UTC m=+2808.757613407" Nov 01 21:17:35.438274 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:35.438221 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:35.439867 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:35.439846 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bb8j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 21:17:35.439931 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:35.439892 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 21:17:36.438532 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:36.438497 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bb8j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 21:17:36.438944 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:36.438552 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 21:17:36.564836 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:17:36.564797 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:17:37.438553 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:37.438511 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bb8j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 21:17:37.438969 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:37.438581 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 21:17:38.438592 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:38.438549 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bb8j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 21:17:38.439022 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:38.438623 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 21:17:39.438604 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:39.438552 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bb8j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 21:17:39.439036 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:39.438623 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 21:17:40.437999 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:40.437915 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bb8j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 21:17:40.437999 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:40.437973 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 21:17:42.116206 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:42.116149 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:42.116894 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:42.116874 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:17:42.117665 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:17:42.117642 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:18:06.551879 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:18:06.551797 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:18:27.262414 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:27.262382 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:18:36.552536 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:18:36.552499 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:18:48.554837 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.554292 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:18:48.712610 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.712578 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl"] Nov 01 21:18:48.716361 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.716341 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812286 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812257 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812452 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812307 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812452 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812364 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812452 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812436 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812592 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812479 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxq8q\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-kube-api-access-jxq8q\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812592 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812499 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812672 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812627 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90234a69-0de6-4530-9d91-7d28c8a604d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812672 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812657 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.812806 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.812691 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.821163 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.821131 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl"] Nov 01 21:18:48.913714 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913671 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.913940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913728 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.913940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913756 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.913940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913778 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.913940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913799 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.913940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913832 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxq8q\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-kube-api-access-jxq8q\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.913940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913895 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.913940 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913934 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90234a69-0de6-4530-9d91-7d28c8a604d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.914588 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.913961 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.914588 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.914424 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.914732 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.914709 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.914796 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.914775 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.914947 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.914925 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.915201 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.915180 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90234a69-0de6-4530-9d91-7d28c8a604d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.917112 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.917060 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.917418 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.917397 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.973323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.973286 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxq8q\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-kube-api-access-jxq8q\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:48.973628 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:48.973600 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gnfzl\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:49.030613 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:49.030564 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:49.468584 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:49.468559 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl"] Nov 01 21:18:49.470499 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:18:49.470468 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90234a69_0de6_4530_9d91_7d28c8a604d5.slice/crio-386d13a7d70d879bb457fd55e2e06de9f583b91dcbe48dd7450de22e20531fbc WatchSource:0}: Error finding container 386d13a7d70d879bb457fd55e2e06de9f583b91dcbe48dd7450de22e20531fbc: Status 404 returned error can't find the container with id 386d13a7d70d879bb457fd55e2e06de9f583b91dcbe48dd7450de22e20531fbc Nov 01 21:18:49.472954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:49.472909 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:18:49.473031 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:49.473011 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:18:49.473111 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:49.473063 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:18:50.359707 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:50.359666 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" event={"ID":"90234a69-0de6-4530-9d91-7d28c8a604d5","Type":"ContainerStarted","Data":"3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c"} Nov 01 21:18:50.360157 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:50.359726 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" event={"ID":"90234a69-0de6-4530-9d91-7d28c8a604d5","Type":"ContainerStarted","Data":"386d13a7d70d879bb457fd55e2e06de9f583b91dcbe48dd7450de22e20531fbc"} Nov 01 21:18:50.537100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:50.536610 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podStartSLOduration=2.536587991 podStartE2EDuration="2.536587991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:50.535425055 +0000 UTC m=+2884.185476635" watchObservedRunningTime="2026-11-01 21:18:50.536587991 +0000 UTC m=+2884.186639539" Nov 01 21:18:51.033859 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:51.033825 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:18:51.034288 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:51.034261 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 21:18:51.034630 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:51.034597 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 21:18:52.033510 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:52.033471 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 21:18:52.034173 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:52.034138 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 21:18:53.032544 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:53.032501 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 21:18:53.032851 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:53.032819 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 21:18:54.031561 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:54.031522 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 21:18:54.032250 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:54.032214 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 21:18:55.035695 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:55.035193 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 21:18:55.035695 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:55.035263 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 21:18:56.032102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:56.031677 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 21:18:56.032102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:56.031749 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 21:18:57.031873 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:57.031837 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 21:18:57.032316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:57.031900 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 21:18:59.072278 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:59.072229 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:18:59.072815 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:59.072284 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:18:59.729391 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:18:59.729351 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:19:00.113265 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:19:00.113222 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:19:00.113828 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:19:00.113301 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:19:00.117243 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:19:00.117210 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:19:00.117537 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:19:00.117515 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:19:00.130811 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:19:00.130780 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:19:06.556045 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:19:06.556004 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:19:19.635839 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:19:19.635805 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:19:36.447551 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:19:36.447470 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:19:36.552972 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:19:36.552933 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:20:06.554044 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:20:06.553999 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:20:07.130087 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:20:07.130045 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:20:17.757222 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:20:17.757181 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:20:30.558447 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:20:30.558409 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:20:36.553000 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:20:36.552961 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:20:55.632619 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:20:55.632570 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:21:05.264324 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:05.264237 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:21:06.552279 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:21:06.552241 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:21:23.448444 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:23.448410 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:21:35.844881 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:35.844840 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:21:36.552156 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:21:36.552115 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:21:38.052253 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:38.052216 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm"] Nov 01 21:21:38.100725 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:38.100692 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78bba8d8-dthwm"] Nov 01 21:21:38.503743 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:38.503702 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2da6f1-8b9c-45c0-8b46-0abcc45f422a" path="/var/lib/kubelet/pods/2e2da6f1-8b9c-45c0-8b46-0abcc45f422a/volumes" Nov 01 21:21:43.050490 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:43.050448 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr"] Nov 01 21:21:43.099666 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:43.099629 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c3637-xtjdr"] Nov 01 21:21:44.502877 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:44.502838 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e51240-447c-4b20-b2d1-9e5611e9ae67" path="/var/lib/kubelet/pods/96e51240-447c-4b20-b2d1-9e5611e9ae67/volumes" Nov 01 21:21:55.913587 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:21:55.913554 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:22:06.554308 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:22:06.554191 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:22:17.829935 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:22:17.829892 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:22:30.931799 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:22:30.931705 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:22:36.556791 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:22:36.556742 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:22:41.962866 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:22:41.962824 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:22:47.781609 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:22:47.781576 4477 scope.go:122] "RemoveContainer" containerID="167ff944a699d9d9d3c804bc9474aa61325ba5520984eb4873c62c337ad31a4d" Nov 01 21:22:47.790902 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:22:47.790877 4477 scope.go:122] "RemoveContainer" containerID="fc41c40aed2e677e929df89f07d615a65f0ecaeb939a3cce5c45aaffc99a2cba" Nov 01 21:23:04.548809 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:23:04.548777 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:23:06.551807 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:23:06.551772 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:23:14.851921 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:23:14.851879 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:23:29.948830 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:23:29.948784 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:23:36.552403 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:23:36.552366 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:23:52.538670 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:23:52.538629 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:24:03.218130 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:24:03.218024 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:24:06.553209 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:24:06.553168 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:24:36.552996 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:24:36.552954 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:24:50.760537 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:24:50.760501 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:25:01.352001 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:25:01.351960 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:25:06.552794 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:25:06.552761 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:25:26.856695 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:25:26.856599 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:25:36.553039 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:25:36.552993 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:25:45.544981 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:25:45.544946 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:26:06.552263 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:26:06.552221 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:26:11.268674 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:26:11.268633 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:26:36.552670 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:26:36.552626 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:26:40.327267 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:26:40.327233 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:26:59.630903 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:26:59.630868 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:27:06.553336 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:27:06.553290 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:27:21.824128 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:21.824068 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:27:32.437365 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:32.437332 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:27:36.553009 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:27:36.552970 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:27:42.617958 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:42.617921 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g"] Nov 01 21:27:42.618466 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:42.618164 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" podUID="716b0934-7ae5-439b-953c-f55dca1f8a5e" containerName="manager" containerID="cri-o://3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445" gracePeriod=10 Nov 01 21:27:42.882503 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:42.882433 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 21:27:42.971853 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:42.971821 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716b0934-7ae5-439b-953c-f55dca1f8a5e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716b0934-7ae5-439b-953c-f55dca1f8a5e-extensions-socket-volume\") pod \"716b0934-7ae5-439b-953c-f55dca1f8a5e\" (UID: \"716b0934-7ae5-439b-953c-f55dca1f8a5e\") " Nov 01 21:27:42.972023 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:42.971949 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716b0934-7ae5-439b-953c-f55dca1f8a5e-kube-api-access-v455n\" (UniqueName: \"kubernetes.io/projected/716b0934-7ae5-439b-953c-f55dca1f8a5e-kube-api-access-v455n\") pod \"716b0934-7ae5-439b-953c-f55dca1f8a5e\" (UID: \"716b0934-7ae5-439b-953c-f55dca1f8a5e\") " Nov 01 21:27:42.972234 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:42.972211 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716b0934-7ae5-439b-953c-f55dca1f8a5e-extensions-socket-volume" pod "716b0934-7ae5-439b-953c-f55dca1f8a5e" (UID: "716b0934-7ae5-439b-953c-f55dca1f8a5e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:27:42.974086 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:42.974059 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716b0934-7ae5-439b-953c-f55dca1f8a5e-kube-api-access-v455n" pod "716b0934-7ae5-439b-953c-f55dca1f8a5e" (UID: "716b0934-7ae5-439b-953c-f55dca1f8a5e"). InnerVolumeSpecName "kube-api-access-v455n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:27:43.072788 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.072752 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v455n\" (UniqueName: \"kubernetes.io/projected/716b0934-7ae5-439b-953c-f55dca1f8a5e-kube-api-access-v455n\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:43.072788 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.072777 4477 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716b0934-7ae5-439b-953c-f55dca1f8a5e-extensions-socket-volume\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:43.472459 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.472424 4477 generic.go:356] "Generic (PLEG): container finished" podID="716b0934-7ae5-439b-953c-f55dca1f8a5e" containerID="3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445" exitCode=0 Nov 01 21:27:43.472639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.472472 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" Nov 01 21:27:43.472639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.472506 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" event={"ID":"716b0934-7ae5-439b-953c-f55dca1f8a5e","Type":"ContainerDied","Data":"3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445"} Nov 01 21:27:43.472639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.472564 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g" event={"ID":"716b0934-7ae5-439b-953c-f55dca1f8a5e","Type":"ContainerDied","Data":"e2435a86585ff035bf2e36b58a1dfa8cedcc447b5ad732513299c2f2beebaaca"} Nov 01 21:27:43.472639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.472576 4477 scope.go:122] "RemoveContainer" containerID="3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445" Nov 01 21:27:43.485459 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.485440 4477 scope.go:122] "RemoveContainer" containerID="3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445" Nov 01 21:27:43.485747 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:27:43.485728 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445\": container with ID starting with 3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445 not found: ID does not exist" containerID="3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445" Nov 01 21:27:43.485811 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.485768 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445"} err="failed to get container status \"3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445\": rpc error: code = NotFound desc = could not find container \"3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445\": container with ID starting with 3c604d5c59737c4a654d316b508c7d61019054df9a21482bb703aacaf0fa0445 not found: ID does not exist" Nov 01 21:27:43.636411 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.636366 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g"] Nov 01 21:27:43.685928 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:43.685894 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bvn9g"] Nov 01 21:27:44.502229 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:27:44.502192 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716b0934-7ae5-439b-953c-f55dca1f8a5e" path="/var/lib/kubelet/pods/716b0934-7ae5-439b-953c-f55dca1f8a5e/volumes" Nov 01 21:28:06.552645 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:28:06.552609 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:28:36.551572 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:28:36.551481 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:28:46.228810 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.228763 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf"] Nov 01 21:28:46.229505 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.229403 4477 cpu_manager.go:414] "containerMap: removing container" podUID="716b0934-7ae5-439b-953c-f55dca1f8a5e" containerName="manager" Nov 01 21:28:46.229505 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.229426 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716b0934-7ae5-439b-953c-f55dca1f8a5e" containerName="manager" Nov 01 21:28:46.229633 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.229513 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="716b0934-7ae5-439b-953c-f55dca1f8a5e" containerName="manager" Nov 01 21:28:46.232714 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.232691 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:46.286635 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.286605 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf"] Nov 01 21:28:46.333630 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.333600 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sg6qc\"" Nov 01 21:28:46.355806 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.355771 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzj4\" (UniqueName: \"kubernetes.io/projected/dc5df5d0-7965-4cbf-98d2-f235583b0a52-kube-api-access-nqzj4\") pod \"kuadrant-operator-controller-manager-678d87766f-lndlf\" (UID: \"dc5df5d0-7965-4cbf-98d2-f235583b0a52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:46.355961 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.355824 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc5df5d0-7965-4cbf-98d2-f235583b0a52-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lndlf\" (UID: \"dc5df5d0-7965-4cbf-98d2-f235583b0a52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:46.456924 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.456887 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzj4\" (UniqueName: \"kubernetes.io/projected/dc5df5d0-7965-4cbf-98d2-f235583b0a52-kube-api-access-nqzj4\") pod \"kuadrant-operator-controller-manager-678d87766f-lndlf\" (UID: \"dc5df5d0-7965-4cbf-98d2-f235583b0a52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:46.457190 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.456937 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc5df5d0-7965-4cbf-98d2-f235583b0a52-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lndlf\" (UID: \"dc5df5d0-7965-4cbf-98d2-f235583b0a52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:46.457296 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.457279 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc5df5d0-7965-4cbf-98d2-f235583b0a52-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lndlf\" (UID: \"dc5df5d0-7965-4cbf-98d2-f235583b0a52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:46.513625 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.513559 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzj4\" (UniqueName: \"kubernetes.io/projected/dc5df5d0-7965-4cbf-98d2-f235583b0a52-kube-api-access-nqzj4\") pod \"kuadrant-operator-controller-manager-678d87766f-lndlf\" (UID: \"dc5df5d0-7965-4cbf-98d2-f235583b0a52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:46.597676 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.597650 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sg6qc\"" Nov 01 21:28:46.606832 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:46.606811 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:47.086952 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:47.086923 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf"] Nov 01 21:28:47.140778 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:28:47.140746 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5df5d0_7965_4cbf_98d2_f235583b0a52.slice/crio-4e7d062b3cbfce8c4441eaa90fa2354ffaa273ea8d3779536afdbbcfc08c1ad8 WatchSource:0}: Error finding container 4e7d062b3cbfce8c4441eaa90fa2354ffaa273ea8d3779536afdbbcfc08c1ad8: Status 404 returned error can't find the container with id 4e7d062b3cbfce8c4441eaa90fa2354ffaa273ea8d3779536afdbbcfc08c1ad8 Nov 01 21:28:47.813843 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:47.813809 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" event={"ID":"dc5df5d0-7965-4cbf-98d2-f235583b0a52","Type":"ContainerStarted","Data":"a88c00c342961a71ca919414b75e39f2492d50cae3804151d4207d6925901a05"} Nov 01 21:28:47.814324 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:47.813862 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" event={"ID":"dc5df5d0-7965-4cbf-98d2-f235583b0a52","Type":"ContainerStarted","Data":"4e7d062b3cbfce8c4441eaa90fa2354ffaa273ea8d3779536afdbbcfc08c1ad8"} Nov 01 21:28:47.814324 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:47.813906 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:28:48.011565 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:48.011510 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" podStartSLOduration=2.011495779 podStartE2EDuration="2.011495779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:28:47.977173368 +0000 UTC m=+3481.627224911" watchObservedRunningTime="2026-11-01 21:28:48.011495779 +0000 UTC m=+3481.661547323" Nov 01 21:28:58.821972 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:28:58.821939 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lndlf" Nov 01 21:29:06.552530 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:29:06.552490 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:29:08.643113 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:29:08.643061 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:29:25.837033 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:29:25.836999 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:29:35.343582 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:29:35.343542 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:29:36.553208 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:29:36.553170 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:29:54.638269 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:29:54.638230 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:30:06.553858 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:30:06.553817 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:30:11.618960 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:30:11.618922 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:30:32.331258 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:30:32.331218 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:30:36.552425 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:30:36.552388 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:30:57.432468 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:30:57.432428 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:31:06.553022 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:31:06.552984 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:31:25.262443 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:31:25.262408 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:31:36.553354 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:31:36.553263 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:31:46.487616 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:31:46.487572 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:32:00.964639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:32:00.964600 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:32:06.553248 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:32:06.553209 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:32:36.551310 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:32:36.551272 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:32:45.847097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:32:45.847053 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:33:04.242192 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:33:04.242094 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:33:06.607427 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:33:06.607391 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:33:19.628513 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:33:19.628476 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:33:34.651187 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:33:34.651147 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:33:36.610034 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:33:36.609994 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:33:47.012433 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:33:47.012400 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:33:57.731578 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:33:57.731535 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:34:06.552953 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:34:06.552913 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:34:20.733679 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.733640 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:34:20.863727 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.863689 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq"] Nov 01 21:34:20.867256 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.867232 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:20.950505 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.950471 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4447a15-0103-40fd-8e66-8768caa0b66c-openshift-service-ca\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:20.950728 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.950516 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z86t\" (UniqueName: \"kubernetes.io/projected/a4447a15-0103-40fd-8e66-8768caa0b66c-kube-api-access-4z86t\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:20.950728 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.950538 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-metrics-certs\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:20.950864 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.950758 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-maas-api-tls\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:20.971454 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.971425 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cms9v\"" Nov 01 21:34:20.971707 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.971688 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 21:34:20.971707 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.971698 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 21:34:20.973708 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:20.973688 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq"] Nov 01 21:34:21.052180 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.052084 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-maas-api-tls\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.052180 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.052146 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4447a15-0103-40fd-8e66-8768caa0b66c-openshift-service-ca\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.052180 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.052172 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z86t\" (UniqueName: \"kubernetes.io/projected/a4447a15-0103-40fd-8e66-8768caa0b66c-kube-api-access-4z86t\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.052438 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.052189 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-metrics-certs\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.052802 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.052773 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4447a15-0103-40fd-8e66-8768caa0b66c-openshift-service-ca\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.055400 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.055375 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-metrics-certs\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.055627 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.055603 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-maas-api-tls\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.109485 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.109447 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z86t\" (UniqueName: \"kubernetes.io/projected/a4447a15-0103-40fd-8e66-8768caa0b66c-kube-api-access-4z86t\") pod \"maas-api-5f57b8b55f-l9zzq\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.179655 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.179621 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:21.553124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.553100 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq"] Nov 01 21:34:21.555746 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:34:21.555712 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4447a15_0103_40fd_8e66_8768caa0b66c.slice/crio-c50ad37860ca3ee1ebcdf9661575014cf198ff75fa3a9a883b4be5570eefebff WatchSource:0}: Error finding container c50ad37860ca3ee1ebcdf9661575014cf198ff75fa3a9a883b4be5570eefebff: Status 404 returned error can't find the container with id c50ad37860ca3ee1ebcdf9661575014cf198ff75fa3a9a883b4be5570eefebff Nov 01 21:34:21.557117 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.557099 4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:34:21.797439 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:21.797400 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" event={"ID":"a4447a15-0103-40fd-8e66-8768caa0b66c","Type":"ContainerStarted","Data":"c50ad37860ca3ee1ebcdf9661575014cf198ff75fa3a9a883b4be5570eefebff"} Nov 01 21:34:22.805393 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:22.805301 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" event={"ID":"a4447a15-0103-40fd-8e66-8768caa0b66c","Type":"ContainerStarted","Data":"112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc"} Nov 01 21:34:22.805393 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:22.805381 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:22.920456 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:22.920402 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" podStartSLOduration=2.057463345 podStartE2EDuration="2.920386293s" totalImagesPullingTime="862.922948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:22.918910888 +0000 UTC m=+3816.568962451" watchObservedRunningTime="2026-11-01 21:34:22.920386293 +0000 UTC m=+3816.570437838" Nov 01 21:34:28.817188 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:28.817101 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:28.851429 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:28.851393 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq"] Nov 01 21:34:28.851607 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:28.851575 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" podUID="a4447a15-0103-40fd-8e66-8768caa0b66c" containerName="maas-api" containerID="cri-o://112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc" gracePeriod=30 Nov 01 21:34:28.901269 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:28.901238 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp"] Nov 01 21:34:28.904837 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:28.904819 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:28.983902 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:28.983871 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp"] Nov 01 21:34:29.030405 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.030371 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d78ec290-9bd8-4475-ba2e-ead644137c63-maas-api-tls\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.030593 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.030418 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d78ec290-9bd8-4475-ba2e-ead644137c63-metrics-certs\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.030667 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.030634 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ec290-9bd8-4475-ba2e-ead644137c63-openshift-service-ca\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.030726 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.030692 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnrp\" (UniqueName: \"kubernetes.io/projected/d78ec290-9bd8-4475-ba2e-ead644137c63-kube-api-access-mlnrp\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.118688 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.118660 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:29.131492 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.131463 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ec290-9bd8-4475-ba2e-ead644137c63-openshift-service-ca\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.131645 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.131505 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnrp\" (UniqueName: \"kubernetes.io/projected/d78ec290-9bd8-4475-ba2e-ead644137c63-kube-api-access-mlnrp\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.131645 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.131537 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d78ec290-9bd8-4475-ba2e-ead644137c63-maas-api-tls\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.131645 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.131572 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d78ec290-9bd8-4475-ba2e-ead644137c63-metrics-certs\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.132221 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.132197 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ec290-9bd8-4475-ba2e-ead644137c63-openshift-service-ca\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.134239 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.134218 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d78ec290-9bd8-4475-ba2e-ead644137c63-maas-api-tls\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.134306 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.134251 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d78ec290-9bd8-4475-ba2e-ead644137c63-metrics-certs\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.190225 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.190190 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnrp\" (UniqueName: \"kubernetes.io/projected/d78ec290-9bd8-4475-ba2e-ead644137c63-kube-api-access-mlnrp\") pod \"maas-api-fb7d7759-qtjlp\" (UID: \"d78ec290-9bd8-4475-ba2e-ead644137c63\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.232299 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.232260 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4447a15-0103-40fd-8e66-8768caa0b66c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4447a15-0103-40fd-8e66-8768caa0b66c-openshift-service-ca\") pod \"a4447a15-0103-40fd-8e66-8768caa0b66c\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " Nov 01 21:34:29.232497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.232328 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4447a15-0103-40fd-8e66-8768caa0b66c-kube-api-access-4z86t\" (UniqueName: \"kubernetes.io/projected/a4447a15-0103-40fd-8e66-8768caa0b66c-kube-api-access-4z86t\") pod \"a4447a15-0103-40fd-8e66-8768caa0b66c\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " Nov 01 21:34:29.232497 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.232437 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-metrics-certs\") pod \"a4447a15-0103-40fd-8e66-8768caa0b66c\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " Nov 01 21:34:29.232631 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.232505 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-maas-api-tls\") pod \"a4447a15-0103-40fd-8e66-8768caa0b66c\" (UID: \"a4447a15-0103-40fd-8e66-8768caa0b66c\") " Nov 01 21:34:29.232693 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.232662 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4447a15-0103-40fd-8e66-8768caa0b66c-openshift-service-ca" pod "a4447a15-0103-40fd-8e66-8768caa0b66c" (UID: "a4447a15-0103-40fd-8e66-8768caa0b66c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:34:29.232889 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.232843 4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4447a15-0103-40fd-8e66-8768caa0b66c-openshift-service-ca\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:29.234528 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.234501 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:29.234924 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.234893 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-metrics-certs" pod "a4447a15-0103-40fd-8e66-8768caa0b66c" (UID: "a4447a15-0103-40fd-8e66-8768caa0b66c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:34:29.235045 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.234893 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4447a15-0103-40fd-8e66-8768caa0b66c-kube-api-access-4z86t" pod "a4447a15-0103-40fd-8e66-8768caa0b66c" (UID: "a4447a15-0103-40fd-8e66-8768caa0b66c"). InnerVolumeSpecName "kube-api-access-4z86t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:34:29.235045 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.234962 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-maas-api-tls" pod "a4447a15-0103-40fd-8e66-8768caa0b66c" (UID: "a4447a15-0103-40fd-8e66-8768caa0b66c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:34:29.333982 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.333947 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z86t\" (UniqueName: \"kubernetes.io/projected/a4447a15-0103-40fd-8e66-8768caa0b66c-kube-api-access-4z86t\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:29.333982 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.333981 4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-metrics-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:29.334241 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.333992 4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4447a15-0103-40fd-8e66-8768caa0b66c-maas-api-tls\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:29.612460 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.612435 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp"] Nov 01 21:34:29.614541 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:34:29.614510 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78ec290_9bd8_4475_ba2e_ead644137c63.slice/crio-03f1adc40b3befc2e2f087254be569363044172a4e87b880bf541a074c16ad84 WatchSource:0}: Error finding container 03f1adc40b3befc2e2f087254be569363044172a4e87b880bf541a074c16ad84: Status 404 returned error can't find the container with id 03f1adc40b3befc2e2f087254be569363044172a4e87b880bf541a074c16ad84 Nov 01 21:34:29.845399 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.845359 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" event={"ID":"d78ec290-9bd8-4475-ba2e-ead644137c63","Type":"ContainerStarted","Data":"03f1adc40b3befc2e2f087254be569363044172a4e87b880bf541a074c16ad84"} Nov 01 21:34:29.846716 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.846691 4477 generic.go:356] "Generic (PLEG): container finished" podID="a4447a15-0103-40fd-8e66-8768caa0b66c" containerID="112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc" exitCode=0 Nov 01 21:34:29.846844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.846733 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" Nov 01 21:34:29.846844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.846764 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" event={"ID":"a4447a15-0103-40fd-8e66-8768caa0b66c","Type":"ContainerDied","Data":"112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc"} Nov 01 21:34:29.846844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.846823 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq" event={"ID":"a4447a15-0103-40fd-8e66-8768caa0b66c","Type":"ContainerDied","Data":"c50ad37860ca3ee1ebcdf9661575014cf198ff75fa3a9a883b4be5570eefebff"} Nov 01 21:34:29.846951 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.846849 4477 scope.go:122] "RemoveContainer" containerID="112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc" Nov 01 21:34:29.856918 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.856754 4477 scope.go:122] "RemoveContainer" containerID="112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc" Nov 01 21:34:29.857137 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:34:29.857117 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc\": container with ID starting with 112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc not found: ID does not exist" containerID="112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc" Nov 01 21:34:29.857285 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:29.857148 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc"} err="failed to get container status \"112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc\": rpc error: code = NotFound desc = could not find container \"112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc\": container with ID starting with 112faa8239d58d107d9690d86bde473eb9c210be73908ba056f8ef70f3c840fc not found: ID does not exist" Nov 01 21:34:30.009742 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:30.009710 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq"] Nov 01 21:34:30.014323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:30.014293 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-l9zzq"] Nov 01 21:34:30.504028 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:30.504008 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4447a15-0103-40fd-8e66-8768caa0b66c" path="/var/lib/kubelet/pods/a4447a15-0103-40fd-8e66-8768caa0b66c/volumes" Nov 01 21:34:30.853346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:30.853306 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" event={"ID":"d78ec290-9bd8-4475-ba2e-ead644137c63","Type":"ContainerStarted","Data":"7c0f353e1db7c30ed21bc5b0e791385665313b19c40bf16a0a6d8ff7d3e6ee4e"} Nov 01 21:34:30.853855 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:30.853396 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:30.969138 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:30.969083 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" podStartSLOduration=2.132740434 podStartE2EDuration="2.969053416s" totalImagesPullingTime="836.312982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:30.966821088 +0000 UTC m=+3824.616872673" watchObservedRunningTime="2026-11-01 21:34:30.969053416 +0000 UTC m=+3824.619104962" Nov 01 21:34:36.552456 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:34:36.552422 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:34:36.865316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:36.865289 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fb7d7759-qtjlp" Nov 01 21:34:44.067688 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:34:44.067651 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:35:06.553581 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:35:06.553541 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:35:28.916687 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:35:28.916638 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:35:36.552308 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:35:36.552270 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:35:49.136165 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:35:49.136125 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:36:06.552866 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:36:06.552781 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:36:36.552788 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:36:36.552750 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:37:06.552863 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:37:06.552817 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:37:36.552811 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:37:36.552694 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:38:06.551855 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:38:06.551813 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:38:36.553309 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:38:36.553273 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:39:06.552945 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:39:06.552850 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:39:27.922299 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:27.922265 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl"] Nov 01 21:39:27.922843 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:27.922490 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" containerID="cri-o://3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c" gracePeriod=30 Nov 01 21:39:27.971808 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:27.971770 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6"] Nov 01 21:39:27.972038 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:27.972002 4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" containerID="cri-o://641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167" gracePeriod=30 Nov 01 21:39:30.120015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:30.119981 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gnfzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:39:30.120434 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:30.120040 4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:39:33.228947 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.228922 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:39:33.257903 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.257882 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:39:33.300417 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300388 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-envoy\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.300544 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300431 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-data\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.300544 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300472 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-kube-api-access-jxq8q\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-kube-api-access-jxq8q\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.300544 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300534 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-certs\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.300721 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300676 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90234a69-0de6-4530-9d91-7d28c8a604d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90234a69-0de6-4530-9d91-7d28c8a604d5-istiod-ca-cert\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.300781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300716 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-credential-socket\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.300781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300744 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-data" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.300781 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300758 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-socket\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.300938 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300790 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-token\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-token\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.300938 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300847 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-certs" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.300938 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300906 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-podinfo\") pod \"90234a69-0de6-4530-9d91-7d28c8a604d5\" (UID: \"90234a69-0de6-4530-9d91-7d28c8a604d5\") " Nov 01 21:39:33.301119 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300959 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-credential-socket" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.301119 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.300996 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-socket" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.301322 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.301296 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.301322 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.301326 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.301658 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.301341 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.301658 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.301355 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.301658 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.301348 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90234a69-0de6-4530-9d91-7d28c8a604d5-istiod-ca-cert" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:39:33.303110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.303088 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-token" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.303199 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.303154 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-podinfo" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:39:33.303277 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.303261 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-kube-api-access-jxq8q" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "kube-api-access-jxq8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.303328 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.303272 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-envoy" pod "90234a69-0de6-4530-9d91-7d28c8a604d5" (UID: "90234a69-0de6-4530-9d91-7d28c8a604d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.402342 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402314 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-token\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-token\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402533 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402385 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a515b52-f936-4e88-ad17-a49ef4c2451c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a515b52-f936-4e88-ad17-a49ef4c2451c-istiod-ca-cert\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402533 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402457 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-podinfo\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402533 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402505 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-data\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402715 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402542 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-kube-api-access-4fw28\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-kube-api-access-4fw28\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402715 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402571 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-socket\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402715 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402603 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-certs\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402715 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402680 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-credential-socket\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402715 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402710 4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-envoy\") pod \"0a515b52-f936-4e88-ad17-a49ef4c2451c\" (UID: \"0a515b52-f936-4e88-ad17-a49ef4c2451c\") " Nov 01 21:39:33.402986 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402829 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-socket" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.402986 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402858 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a515b52-f936-4e88-ad17-a49ef4c2451c-istiod-ca-cert" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:39:33.403114 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.402991 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-certs" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.403220 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403202 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90234a69-0de6-4530-9d91-7d28c8a604d5-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.403289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403226 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.403289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403240 4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a515b52-f936-4e88-ad17-a49ef4c2451c-istiod-ca-cert\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.403289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403255 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.403289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403269 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90234a69-0de6-4530-9d91-7d28c8a604d5-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.403289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403284 4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.403488 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403298 4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-workload-certs\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.403488 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403312 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxq8q\" (UniqueName: \"kubernetes.io/projected/90234a69-0de6-4530-9d91-7d28c8a604d5-kube-api-access-jxq8q\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.403488 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403245 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-data" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.403771 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.403729 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-credential-socket" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.404896 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.404863 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-token" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.405023 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.404869 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-podinfo" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:39:33.405281 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.405265 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-envoy" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.405378 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.405362 4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-kube-api-access-4fw28" pod "0a515b52-f936-4e88-ad17-a49ef4c2451c" (UID: "0a515b52-f936-4e88-ad17-a49ef4c2451c"). InnerVolumeSpecName "kube-api-access-4fw28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.453327 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.453300 4477 generic.go:356] "Generic (PLEG): container finished" podID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerID="641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167" exitCode=0 Nov 01 21:39:33.453440 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.453364 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" Nov 01 21:39:33.453440 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.453384 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" event={"ID":"0a515b52-f936-4e88-ad17-a49ef4c2451c","Type":"ContainerDied","Data":"641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167"} Nov 01 21:39:33.453440 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.453435 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6" event={"ID":"0a515b52-f936-4e88-ad17-a49ef4c2451c","Type":"ContainerDied","Data":"dd1a95ff316b496ad2c0d384350b2408be3d6eadc6a3bf188c6af5c5dc4c9006"} Nov 01 21:39:33.453567 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.453466 4477 scope.go:122] "RemoveContainer" containerID="641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167" Nov 01 21:39:33.454867 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.454849 4477 generic.go:356] "Generic (PLEG): container finished" podID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerID="3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c" exitCode=0 Nov 01 21:39:33.454980 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.454890 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" event={"ID":"90234a69-0de6-4530-9d91-7d28c8a604d5","Type":"ContainerDied","Data":"3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c"} Nov 01 21:39:33.454980 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.454944 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" event={"ID":"90234a69-0de6-4530-9d91-7d28c8a604d5","Type":"ContainerDied","Data":"386d13a7d70d879bb457fd55e2e06de9f583b91dcbe48dd7450de22e20531fbc"} Nov 01 21:39:33.454980 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.454903 4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl" Nov 01 21:39:33.464435 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.464400 4477 scope.go:122] "RemoveContainer" containerID="641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167" Nov 01 21:39:33.464753 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:39:33.464734 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167\": container with ID starting with 641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167 not found: ID does not exist" containerID="641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167" Nov 01 21:39:33.464857 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.464762 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167"} err="failed to get container status \"641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167\": rpc error: code = NotFound desc = could not find container \"641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167\": container with ID starting with 641030e689ae8584837d85b616b911038eecbf9fdc17a6a66dfe88976195c167 not found: ID does not exist" Nov 01 21:39:33.464857 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.464780 4477 scope.go:122] "RemoveContainer" containerID="3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c" Nov 01 21:39:33.473718 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.473695 4477 scope.go:122] "RemoveContainer" containerID="3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c" Nov 01 21:39:33.474017 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:39:33.473997 4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c\": container with ID starting with 3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c not found: ID does not exist" containerID="3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c" Nov 01 21:39:33.474132 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.474024 4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c"} err="failed to get container status \"3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c\": rpc error: code = NotFound desc = could not find container \"3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c\": container with ID starting with 3b33c10bbc83c779057717dcea0db3aa365b7d618a9ba07917bcac3bfb6b851c not found: ID does not exist" Nov 01 21:39:33.503917 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.503887 4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-credential-socket\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.503917 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.503916 4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-envoy\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.504145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.503930 4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-token\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.504145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.503941 4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-podinfo\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.504145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.503953 4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a515b52-f936-4e88-ad17-a49ef4c2451c-istio-data\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.504145 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.503965 4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fw28\" (UniqueName: \"kubernetes.io/projected/0a515b52-f936-4e88-ad17-a49ef4c2451c-kube-api-access-4fw28\") on node \"ip-10-0-133-242.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.620582 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.620547 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6"] Nov 01 21:39:33.622844 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.622822 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bb8j6"] Nov 01 21:39:33.730623 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.730543 4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl"] Nov 01 21:39:33.733515 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:33.733490 4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gnfzl"] Nov 01 21:39:34.503423 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:34.503388 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" path="/var/lib/kubelet/pods/0a515b52-f936-4e88-ad17-a49ef4c2451c/volumes" Nov 01 21:39:34.503988 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:39:34.503967 4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" path="/var/lib/kubelet/pods/90234a69-0de6-4530-9d91-7d28c8a604d5/volumes" Nov 01 21:39:36.552099 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:39:36.552042 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:40:06.553735 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:40:06.553693 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:40:28.752877 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:28.752790 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:40:36.554363 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:40:36.554317 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:40:36.943947 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:36.943911 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:40:48.051954 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:48.051921 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:40:58.614060 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614019 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9"] Nov 01 21:40:58.614684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614550 4477 cpu_manager.go:414] "containerMap: removing container" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" Nov 01 21:40:58.614684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614567 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" Nov 01 21:40:58.614684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614586 4477 cpu_manager.go:414] "containerMap: removing container" podUID="a4447a15-0103-40fd-8e66-8768caa0b66c" containerName="maas-api" Nov 01 21:40:58.614684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614591 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4447a15-0103-40fd-8e66-8768caa0b66c" containerName="maas-api" Nov 01 21:40:58.614684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614606 4477 cpu_manager.go:414] "containerMap: removing container" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" Nov 01 21:40:58.614684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614610 4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" Nov 01 21:40:58.614684 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614687 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a515b52-f936-4e88-ad17-a49ef4c2451c" containerName="istio-proxy" Nov 01 21:40:58.614937 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614699 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4447a15-0103-40fd-8e66-8768caa0b66c" containerName="maas-api" Nov 01 21:40:58.614937 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.614706 4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="90234a69-0de6-4530-9d91-7d28c8a604d5" containerName="istio-proxy" Nov 01 21:40:58.617782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.617766 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.676751 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.676713 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.676930 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.676777 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.676930 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.676864 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ab0222-e992-4c13-ae34-aca5127b28c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.677015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.676972 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.677015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.677004 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.677153 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.677137 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.677195 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.677165 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9fl\" (UniqueName: \"kubernetes.io/projected/c5ab0222-e992-4c13-ae34-aca5127b28c5-kube-api-access-8l9fl\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.677240 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.677195 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.677291 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.677243 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.778519 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778466 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.778519 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778522 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.778782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778588 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.778782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778617 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9fl\" (UniqueName: \"kubernetes.io/projected/c5ab0222-e992-4c13-ae34-aca5127b28c5-kube-api-access-8l9fl\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.778782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778648 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.778782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778673 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.778782 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778748 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.779035 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778821 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.779035 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778863 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ab0222-e992-4c13-ae34-aca5127b28c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.779035 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.778916 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.779035 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.779002 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.779248 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.779173 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.779368 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.779347 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.779567 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.779551 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ab0222-e992-4c13-ae34-aca5127b28c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.781131 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.781104 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.781289 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.781268 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.817159 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.817126 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9"] Nov 01 21:40:58.836813 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.836778 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ab0222-e992-4c13-ae34-aca5127b28c5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.836989 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.836965 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9fl\" (UniqueName: \"kubernetes.io/projected/c5ab0222-e992-4c13-ae34-aca5127b28c5-kube-api-access-8l9fl\") pod \"maas-default-gateway-openshift-default-58c6585447-vmxr9\" (UID: \"c5ab0222-e992-4c13-ae34-aca5127b28c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:58.931238 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:58.931139 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:40:59.357707 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:59.357683 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9"] Nov 01 21:40:59.359718 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:40:59.359695 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ab0222_e992_4c13_ae34_aca5127b28c5.slice/crio-a916680074b5d66280b96e7e4b5cd5fe2d5569cc54f8b002509d536051b81827 WatchSource:0}: Error finding container a916680074b5d66280b96e7e4b5cd5fe2d5569cc54f8b002509d536051b81827: Status 404 returned error can't find the container with id a916680074b5d66280b96e7e4b5cd5fe2d5569cc54f8b002509d536051b81827 Nov 01 21:40:59.361851 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:59.361816 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:40:59.361955 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:59.361897 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:40:59.361955 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:59.361945 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:40:59.907868 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:59.907831 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" event={"ID":"c5ab0222-e992-4c13-ae34-aca5127b28c5","Type":"ContainerStarted","Data":"4374de10857ab28fbed43260a3c29f7974ed8ab0fef6fc67b9df68224c018949"} Nov 01 21:40:59.908361 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:59.907886 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" event={"ID":"c5ab0222-e992-4c13-ae34-aca5127b28c5","Type":"ContainerStarted","Data":"a916680074b5d66280b96e7e4b5cd5fe2d5569cc54f8b002509d536051b81827"} Nov 01 21:40:59.937250 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:40:59.937203 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:41:00.037100 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:00.026779 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" podStartSLOduration=2.026758436 podStartE2EDuration="2.026758436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:00.024400278 +0000 UTC m=+4213.674451849" watchObservedRunningTime="2026-11-01 21:41:00.026758436 +0000 UTC m=+4213.676809982" Nov 01 21:41:00.937309 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:00.937275 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vmxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:41:00.938212 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:00.938161 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" podUID="c5ab0222-e992-4c13-ae34-aca5127b28c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 21:41:01.938636 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:01.938599 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vmxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:41:01.939056 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:01.938655 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" podUID="c5ab0222-e992-4c13-ae34-aca5127b28c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 21:41:02.936102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:02.935508 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vmxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:41:02.936102 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:02.935573 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" podUID="c5ab0222-e992-4c13-ae34-aca5127b28c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 21:41:03.934099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:03.933556 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vmxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:41:03.934099 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:03.933624 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" podUID="c5ab0222-e992-4c13-ae34-aca5127b28c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 21:41:04.933638 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:04.933593 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vmxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:41:04.933949 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:04.933915 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" podUID="c5ab0222-e992-4c13-ae34-aca5127b28c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 21:41:05.932318 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:05.932274 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vmxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:41:05.932765 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:05.932337 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" podUID="c5ab0222-e992-4c13-ae34-aca5127b28c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 21:41:06.554016 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:41:06.553971 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:41:06.932547 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:06.932508 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vmxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:41:06.933010 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:06.932580 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" podUID="c5ab0222-e992-4c13-ae34-aca5127b28c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 21:41:07.935926 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:07.935895 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:41:07.936464 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:07.936305 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:41:07.936888 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:07.936867 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vmxr9" Nov 01 21:41:08.845323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:08.845281 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:41:13.735927 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.735890 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf"] Nov 01 21:41:13.739754 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.739734 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847004 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.846971 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7m7\" (UniqueName: \"kubernetes.io/projected/c65cb197-db69-46e2-a3d5-97212f7452c8-kube-api-access-th7m7\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847192 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.847014 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847192 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.847106 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847192 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.847147 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847192 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.847177 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847346 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.847293 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847391 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.847354 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847391 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.847382 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.847510 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.847417 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65cb197-db69-46e2-a3d5-97212f7452c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.937866 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.937832 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf"] Nov 01 21:41:13.948949 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.948907 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949129 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.948999 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949129 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949099 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65cb197-db69-46e2-a3d5-97212f7452c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949262 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949165 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7m7\" (UniqueName: \"kubernetes.io/projected/c65cb197-db69-46e2-a3d5-97212f7452c8-kube-api-access-th7m7\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949262 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949224 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949381 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949263 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949381 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949318 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949496 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949383 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949496 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949392 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949496 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949441 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949496 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949448 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.949852 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949829 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.950023 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.949998 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.950576 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.950555 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65cb197-db69-46e2-a3d5-97212f7452c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.951841 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.951821 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:13.952583 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:13.952566 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:14.006252 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:14.006174 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65cb197-db69-46e2-a3d5-97212f7452c8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:14.006560 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:14.006533 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7m7\" (UniqueName: \"kubernetes.io/projected/c65cb197-db69-46e2-a3d5-97212f7452c8-kube-api-access-th7m7\") pod \"maas-default-gateway-openshift-default-58c6585447-cfgkf\" (UID: \"c65cb197-db69-46e2-a3d5-97212f7452c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:14.052951 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:14.052895 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:14.489015 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:14.488988 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf"] Nov 01 21:41:14.490565 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:41:14.490539 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65cb197_db69_46e2_a3d5_97212f7452c8.slice/crio-d09b2f7642ffc2827fb2a9fecd665995df1c7fbb4a9019dd2e0782dfc15267ae WatchSource:0}: Error finding container d09b2f7642ffc2827fb2a9fecd665995df1c7fbb4a9019dd2e0782dfc15267ae: Status 404 returned error can't find the container with id d09b2f7642ffc2827fb2a9fecd665995df1c7fbb4a9019dd2e0782dfc15267ae Nov 01 21:41:14.492988 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:14.492946 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:41:14.493103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:14.493057 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:41:14.493157 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:14.493138 4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 21:41:15.043108 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:15.039759 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" event={"ID":"c65cb197-db69-46e2-a3d5-97212f7452c8","Type":"ContainerStarted","Data":"a9b3ba30b08fa12bc7d272671281ee686ae056d0e794a4a8efedf1ffbf41260f"} Nov 01 21:41:15.043108 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:15.039824 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" event={"ID":"c65cb197-db69-46e2-a3d5-97212f7452c8","Type":"ContainerStarted","Data":"d09b2f7642ffc2827fb2a9fecd665995df1c7fbb4a9019dd2e0782dfc15267ae"} Nov 01 21:41:15.058109 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:15.056408 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:15.064097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:15.061212 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cfgkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:41:15.064097 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:15.061266 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podUID="c65cb197-db69-46e2-a3d5-97212f7452c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 21:41:15.207103 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:15.206450 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podStartSLOduration=2.206429911 podStartE2EDuration="2.206429911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:15.205269031 +0000 UTC m=+4228.855320576" watchObservedRunningTime="2026-11-01 21:41:15.206429911 +0000 UTC m=+4228.856481459" Nov 01 21:41:16.059117 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:16.054502 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cfgkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:41:16.059117 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:16.054562 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podUID="c65cb197-db69-46e2-a3d5-97212f7452c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 21:41:17.053305 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:17.053273 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cfgkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:41:17.053490 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:17.053322 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podUID="c65cb197-db69-46e2-a3d5-97212f7452c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 21:41:18.053326 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:18.053288 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cfgkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:41:18.053738 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:18.053347 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podUID="c65cb197-db69-46e2-a3d5-97212f7452c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 21:41:19.053659 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:19.053614 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cfgkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:41:19.054110 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:19.053688 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podUID="c65cb197-db69-46e2-a3d5-97212f7452c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 21:41:20.053225 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:20.053194 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cfgkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:41:20.053396 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:20.053243 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podUID="c65cb197-db69-46e2-a3d5-97212f7452c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 21:41:21.053700 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:21.053663 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cfgkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:41:21.054133 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:21.053719 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podUID="c65cb197-db69-46e2-a3d5-97212f7452c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 21:41:22.053374 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:22.053341 4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cfgkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:41:22.053558 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:22.053415 4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" podUID="c65cb197-db69-46e2-a3d5-97212f7452c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 21:41:23.057147 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:23.057114 4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:23.057628 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:23.057507 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:23.058063 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:23.058047 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cfgkf" Nov 01 21:41:36.551873 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:41:36.551832 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:41:39.651669 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:39.651634 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k2jc8"] Nov 01 21:41:49.400163 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:49.400131 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g52pk_fa08a527-ddee-4c0f-866b-8e5d6ce4dfef/manager/0.log" Nov 01 21:41:49.936694 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:49.936661 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m74c5_b3017bf6-3e29-4780-9053-6853d3329613/manager/0.log" Nov 01 21:41:50.813439 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:50.813416 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hkzfs_583e8cf3-8a77-46f3-85d8-40f64d4db713/server/0.log" Nov 01 21:41:52.484392 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:52.484360 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fb7d7759-qtjlp_d78ec290-9bd8-4475-ba2e-ead644137c63/maas-api/0.log" Nov 01 21:41:53.424472 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:53.424443 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bbjnp_310295d3-536c-484b-acb6-757d91892c4a/postgres/0.log" Nov 01 21:41:55.304459 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:55.304429 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kcx4j_d41f48c6-2ec9-47f8-b009-b6bc66d9288e/manager/0.log" Nov 01 21:41:55.615793 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:55.615766 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-255lc_1e6e09bf-8759-4f40-931d-fcd5d73146c9/manager/0.log" Nov 01 21:41:56.242025 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:56.241992 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lndlf_dc5df5d0-7965-4cbf-98d2-f235583b0a52/manager/0.log" Nov 01 21:41:56.609575 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:56.609503 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-k2jc8_2d9aff2d-e468-4399-ba75-ee34150146ba/limitador/0.log" Nov 01 21:41:56.834047 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:56.834012 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4cbs2_4da15d47-468c-483b-8047-2de04909f3fb/manager/0.log" Nov 01 21:41:57.594062 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:57.594018 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt_1bb50d1b-3ccf-4e41-8caf-3be2712e4c19/istio-proxy/0.log" Nov 01 21:41:59.409041 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:59.409012 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-cfgkf_c65cb197-db69-46e2-a3d5-97212f7452c8/istio-proxy/0.log" Nov 01 21:41:59.675717 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:41:59.675614 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-k7ndz_c642c26c-c492-4b0b-9c1b-9a7cb89e7d97/istio-proxy/0.log" Nov 01 21:42:00.640706 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:00.640669 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-vmxr9_c5ab0222-e992-4c13-ae34-aca5127b28c5/istio-proxy/0.log" Nov 01 21:42:00.906101 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:00.905996 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq_854e0759-b495-4e67-8bae-b96b8c77ef36/istio-proxy/0.log" Nov 01 21:42:06.552236 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:42:06.552201 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:42:13.806761 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:13.806727 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g52pk_fa08a527-ddee-4c0f-866b-8e5d6ce4dfef/manager/0.log" Nov 01 21:42:14.349385 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:14.349352 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m74c5_b3017bf6-3e29-4780-9053-6853d3329613/manager/0.log" Nov 01 21:42:15.055389 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:15.055360 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hkzfs_583e8cf3-8a77-46f3-85d8-40f64d4db713/server/0.log" Nov 01 21:42:16.273622 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:16.273595 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fb7d7759-qtjlp_d78ec290-9bd8-4475-ba2e-ead644137c63/maas-api/0.log" Nov 01 21:42:17.232541 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:17.232506 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bbjnp_310295d3-536c-484b-acb6-757d91892c4a/postgres/0.log" Nov 01 21:42:18.789559 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:18.789529 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kcx4j_d41f48c6-2ec9-47f8-b009-b6bc66d9288e/manager/0.log" Nov 01 21:42:19.092331 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:19.092309 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-255lc_1e6e09bf-8759-4f40-931d-fcd5d73146c9/manager/0.log" Nov 01 21:42:19.530096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:19.529972 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lndlf_dc5df5d0-7965-4cbf-98d2-f235583b0a52/manager/0.log" Nov 01 21:42:19.925639 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:19.925610 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-k2jc8_2d9aff2d-e468-4399-ba75-ee34150146ba/limitador/0.log" Nov 01 21:42:20.132474 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:20.132428 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4cbs2_4da15d47-468c-483b-8047-2de04909f3fb/manager/0.log" Nov 01 21:42:20.708792 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:20.708746 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt_1bb50d1b-3ccf-4e41-8caf-3be2712e4c19/istio-proxy/0.log" Nov 01 21:42:22.576088 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:22.576044 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-cfgkf_c65cb197-db69-46e2-a3d5-97212f7452c8/istio-proxy/0.log" Nov 01 21:42:22.827672 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:22.827594 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-k7ndz_c642c26c-c492-4b0b-9c1b-9a7cb89e7d97/istio-proxy/0.log" Nov 01 21:42:23.754675 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:23.754646 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-vmxr9_c5ab0222-e992-4c13-ae34-aca5127b28c5/istio-proxy/0.log" Nov 01 21:42:24.014065 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:42:24.013988 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-gvjgq_854e0759-b495-4e67-8bae-b96b8c77ef36/istio-proxy/0.log" Nov 01 21:42:36.552173 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:42:36.552135 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:43:06.552586 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:43:06.552549 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:43:36.551959 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:43:36.551879 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:43:39.376646 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:39.376606 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-658ds_689471e9-1161-4ae7-b08a-15b85ffaa359/global-pull-secret-syncer/0.log" Nov 01 21:43:42.964314 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:42.964283 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9msm2_c481a350-2dc5-450b-b59c-2a37640d458e/konnectivity-agent/0.log" Nov 01 21:43:44.324979 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:44.324945 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-242.us-west-1.compute.internal_fb754e3320d8dad235549d1c4b2afdd3/haproxy/0.log" Nov 01 21:43:49.451699 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.451665 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kcx4j_d41f48c6-2ec9-47f8-b009-b6bc66d9288e/manager/0.log" Nov 01 21:43:49.736316 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.736244 4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk"] Nov 01 21:43:49.739682 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.739668 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.792180 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.792157 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r24pw\"/\"openshift-service-ca.crt\"" Nov 01 21:43:49.800571 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.800550 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-podres\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.800663 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.800584 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-lib-modules\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.800663 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.800610 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-proc\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.800750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.800661 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwkd\" (UniqueName: \"kubernetes.io/projected/d05005ce-97d8-40ec-becd-b33d0dd0019d-kube-api-access-8xwkd\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.800750 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.800709 4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-sys\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.841268 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.841245 4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r24pw\"/\"default-dockercfg-pp4xf\"" Nov 01 21:43:49.841376 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.841272 4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r24pw\"/\"kube-root-ca.crt\"" Nov 01 21:43:49.843877 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.843854 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk"] Nov 01 21:43:49.867245 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.867218 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-255lc_1e6e09bf-8759-4f40-931d-fcd5d73146c9/manager/0.log" Nov 01 21:43:49.901185 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901159 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-podres\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.901323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901194 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-lib-modules\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.901323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901220 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-proc\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.901323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901237 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwkd\" (UniqueName: \"kubernetes.io/projected/d05005ce-97d8-40ec-becd-b33d0dd0019d-kube-api-access-8xwkd\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.901323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901280 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-podres\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.901323 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901313 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-lib-modules\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.901517 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901313 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-proc\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.901517 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901316 4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-sys\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.901517 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.901372 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05005ce-97d8-40ec-becd-b33d0dd0019d-sys\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:49.957785 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:49.957755 4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwkd\" (UniqueName: \"kubernetes.io/projected/d05005ce-97d8-40ec-becd-b33d0dd0019d-kube-api-access-8xwkd\") pod \"perf-node-gather-daemonset-sq4vk\" (UID: \"d05005ce-97d8-40ec-becd-b33d0dd0019d\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:50.052175 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:50.052111 4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:50.422337 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:50.422307 4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk"] Nov 01 21:43:50.424605 ip-10-0-133-242 kubenswrapper[4477]: W1101 21:43:50.424580 4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd05005ce_97d8_40ec_becd_b33d0dd0019d.slice/crio-f5b393885fd88436e9236f586711aa497206bc3d74c66c81d559e44a638e96f0 WatchSource:0}: Error finding container f5b393885fd88436e9236f586711aa497206bc3d74c66c81d559e44a638e96f0: Status 404 returned error can't find the container with id f5b393885fd88436e9236f586711aa497206bc3d74c66c81d559e44a638e96f0 Nov 01 21:43:50.760152 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:50.760059 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lndlf_dc5df5d0-7965-4cbf-98d2-f235583b0a52/manager/0.log" Nov 01 21:43:50.901064 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:50.901027 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" event={"ID":"d05005ce-97d8-40ec-becd-b33d0dd0019d","Type":"ContainerStarted","Data":"827f5a4693dcf6ab24906346e039c62a474c9dc3bb486666799962803f81bbcc"} Nov 01 21:43:50.901268 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:50.901107 4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" event={"ID":"d05005ce-97d8-40ec-becd-b33d0dd0019d","Type":"ContainerStarted","Data":"f5b393885fd88436e9236f586711aa497206bc3d74c66c81d559e44a638e96f0"} Nov 01 21:43:50.901268 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:50.901254 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:51.016605 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:51.016508 4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" podStartSLOduration=2.016495038 podStartE2EDuration="2.016495038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:51.014033661 +0000 UTC m=+4384.664085206" watchObservedRunningTime="2026-11-01 21:43:51.016495038 +0000 UTC m=+4384.666546582" Nov 01 21:43:51.273727 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:51.273649 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-k2jc8_2d9aff2d-e468-4399-ba75-ee34150146ba/limitador/0.log" Nov 01 21:43:51.731208 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:51.731178 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4cbs2_4da15d47-468c-483b-8047-2de04909f3fb/manager/0.log" Nov 01 21:43:54.349213 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:54.349182 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9d8597c-4293-4179-a45e-1d221615a5b3/alertmanager/0.log" Nov 01 21:43:54.765087 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:54.765010 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9d8597c-4293-4179-a45e-1d221615a5b3/config-reloader/0.log" Nov 01 21:43:55.180198 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:55.180163 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9d8597c-4293-4179-a45e-1d221615a5b3/kube-rbac-proxy-web/0.log" Nov 01 21:43:55.596907 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:55.596879 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9d8597c-4293-4179-a45e-1d221615a5b3/kube-rbac-proxy/0.log" Nov 01 21:43:56.014652 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:56.014577 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9d8597c-4293-4179-a45e-1d221615a5b3/kube-rbac-proxy-metric/0.log" Nov 01 21:43:56.432284 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:56.432253 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9d8597c-4293-4179-a45e-1d221615a5b3/prom-label-proxy/0.log" Nov 01 21:43:56.849096 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:56.849050 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9d8597c-4293-4179-a45e-1d221615a5b3/init-config-reloader/0.log" Nov 01 21:43:56.917120 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:56.917093 4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-sq4vk" Nov 01 21:43:57.791124 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:57.791091 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ggqkn_2a97282f-8f28-4dfe-9c5d-1fb7a25c9484/kube-state-metrics/0.log" Nov 01 21:43:58.202879 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:58.202849 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ggqkn_2a97282f-8f28-4dfe-9c5d-1fb7a25c9484/kube-rbac-proxy-main/0.log" Nov 01 21:43:58.618636 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:43:58.618610 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ggqkn_2a97282f-8f28-4dfe-9c5d-1fb7a25c9484/kube-rbac-proxy-self/0.log" Nov 01 21:44:01.128556 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:44:01.128528 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j9v7z_1462938c-0e23-476c-acf3-9a7e13acfc99/node-exporter/0.log" Nov 01 21:44:01.544291 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:44:01.544204 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j9v7z_1462938c-0e23-476c-acf3-9a7e13acfc99/kube-rbac-proxy/0.log" Nov 01 21:44:01.959437 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:44:01.959410 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j9v7z_1462938c-0e23-476c-acf3-9a7e13acfc99/init-textfile/0.log" Nov 01 21:44:06.552847 ip-10-0-133-242 kubenswrapper[4477]: E1101 21:44:06.552792 4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xtldx" containerName="extractor" Nov 01 21:44:07.813295 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:44:07.813264 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-tn6bv_437a2f7a-e562-480f-a73c-5c8ea695a375/prometheus-operator/0.log" Nov 01 21:44:08.232315 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:44:08.232240 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-tn6bv_437a2f7a-e562-480f-a73c-5c8ea695a375/kube-rbac-proxy/0.log" Nov 01 21:44:12.466563 ip-10-0-133-242 kubenswrapper[4477]: I1101 21:44:12.466522 4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-4blzt_1bb50d1b-3ccf-4e41-8caf-3be2712e4c19/istio-proxy/0.log"